Implementing the Feedback Functionality in SUSI Web Chat

SUSI AI now has a feedback feature where it collects user’s feedback for every response to learn and improve itself. The first step towards guided learning is building a dataset through a feedback mechanism which can be used to learn from and improvise the skill selection mechanism responsible for answering the user queries.

The flow behind the feedback mechanism is :

  1. For every SUSI response show thumbs up and thumbs down buttons.
  2. For the older messages, the feedback thumbs are disabled and only display the feedback already given. The user cannot change the feedback already given.
  3. For the latest SUSI response the user can change his feedback by clicking on thumbs up if he likes the response, else on thumbs down, until he gives a new query.
  4. When the new query is given by the user, the feedback recorded for the previous response is sent to the server.

Let’s visit SUSI Web Chat and try this out.

We can find the feedback thumbs for the response messages. The user cannot change the feedback he has already given for previous messages. For the latest message the user can toggle feedback until he sends the next query.

How is this implemented?

We first design the UI for feedback thumbs using Material UI SVG Icons. We need a separate component for the feedback UI because we need to store the state of feedback as positive or negative because we are allowing the user to change his feedback for the latest response until a new query is sent. And whenever the user clicks on a thumb, we update the state of the component as positive or negative accordingly.

import ThumbUp from 'material-ui/svg-icons/action/thumb-up';
import ThumbDown from 'material-ui/svg-icons/action/thumb-down';

feedbackButtons = (
  <span className='feedback' style={feedbackStyle}>
    <ThumbUp
      onClick={this.rateSkill.bind(this,'positive')}
      style={feedbackIndicator}
      color={positiveFeedbackColor}/>
    <ThumbDown
      onClick={this.rateSkill.bind(this,'negative')}
      style={feedbackIndicator}
      color={negativeFeedbackColor}/>
  </span>
);

The next step is to store the response in Message Store using saveFeedback Action. This will help us later to send the feedback to the server by querying it from the Message Store. The Action calls the Dispatcher with FEEDBACK_RECEIVED ActionType which is collected in the MessageStore and the feedback is updated in the Message Store.

let feedback = this.state.skill;

if(!(Object.keys(feedback).length === 0 &&    
feedback.constructor === Object)){
  feedback.rating = rating;
  this.props.message.feedback.rating = rating;
  Actions.saveFeedback(feedback);
}

case ActionTypes.FEEDBACK_RECEIVED: {
  _feedback = action.feedback;
  MessageStore.emitChange();
  break;
}

The final step is to send the feedback to the server. The server endpoint to store feedback for a skill requires other parameters apart from feedback to identify the skill. The server response contains an attribute `skills` which gives the path of the skill used to answer that query. From that path we need to parse :

  • Model : Highest level of abstraction for categorising skills
  • Group : Different groups under a model
  • Language : Language of the skill
  • Skill : Name of the skill

For example, for the query `what is the capital of germany` , the skills object is

"skills": ["/susi_skill_data/models/general/smalltalk/en/English-Standalone-aiml2susi.txt"]

So, for this skill,

    • Model : general
    • Group : smalltalk
    • Language : en
    • Skill : English-Standalone-aiml2susi

The server endpoint to store feedback for a particular skill is :

BASE_URL+'/cms/rateSkill.json?model=MODEL&group=GROUP&language=LANGUAGE&skill=SKILL&rating=RATING'

Where Model, Group, Language and Skill are parsed from the skill attribute of server response as discussed above and the Rating is either positive or negative and is collected from the user when he clicks on feedback thumbs.

When a new query is sent, the sendFeedback Action is triggered with the required attributes to make the server call to store feedback on server. The client then makes an Ajax call to the rateSkill endpoint to send the feedback to the server.

let url = BASE_URL+'/cms/rateSkill.json?'+
          'model='+feedback.model+
          '&group='+feedback.group+
          '&language='+feedback.language+
          '&skill='+feedback.skill+
          '&rating='+feedback.rating;

$.ajax({
  url: url,
  dataType: 'jsonp',
  crossDomain: true,
  timeout: 3000,
  async: false,
  success: function (response) {
    console.log(response);
  },
  error: function(errorThrown){
    console.log(errorThrown);
  }
});

This is how the feedback feedback mechanism works in SUSI Web Chat. The entire code can be found at SUSI Web Chat Repository.

Resources

 

Continue ReadingImplementing the Feedback Functionality in SUSI Web Chat

How to Make SUSI Bot for Google Hangouts

Google Hangouts is a platform for communication provided by Google. We can also make bots for Google Hangouts. In fact, in this blog, we will make SUSI bot for Google Hangouts. In order to make it we will follow these steps:

  1. Install Node.js from the link below on your computer if you haven’t installed it already.
    https://nodejs.org/en/
  2. Create a folder with any name and open a shell and change your current directory to the new folder you created.
  3. Type npm init in the shell and enter details like name, version and entry point.
  4. Create a file with the same name that you wrote in entry point in an above-given step. i.app.js and it should be in the same folder you created.
  5. Type following commands in command line  npm install –save hangouts-bot. After hangouts-bot is installed, type npm install –save express.  On success, type npm install –save request. When all the modules are installed check your package.json file; therein you’ll see the modules in the dependencies portion.
  6. Your package.json file should look like this.

    {
     "name": "susi_hangout",
     "version": "1.0.0",
     "description": "SUSI AI bot for Google Hangouts",
     "main": "app.js",
     "scripts": {
       "start": "node app.js"
     },
     "author": "",
     "license": "",
     "dependencies": {
       "express": "^4.15.3",
       "hangouts-bot": "^0.2.1",
       "request": "^2.81.0"
     }
    }
    
  7. Copy following code into file you created i.e index.js
    var hangoutsBot = require("hangouts-bot");
    var bot = new hangoutsBot("your-email-here", process.env.password|| config.password);
    var request = require('request');
    var express = require('express');
    
    var app = express();
    app.set('port', 8080);
    
    bot.on('online', function() {
       console.log('online');
    });
    
    bot.on('message', function(from, message) {
       console.log(from + ">> " + message);
    
           var options1 = {
           method: 'GET',
           url: 'http://api.susi.ai/susi/chat.json',
           qs: {
               timezoneOffset: '-330',
               q: message
           }
       };
    
       request(options1, function(error, response, body) {
               if (error) throw new Error(error);
               // answer fetched from susi
               var type = (JSON.parse(body)).answers[0].actions;
               var answer, columns, data, key, i, count;
               if (type.length == 1 && type[0].type == "answer") {
                   answer = (JSON.parse(body)).answers[0].actions[0].expression;
                   bot.sendMessage(from, answer);
               } else if (type.length == 1 && type[0].type == "table") {
                   data = JSON.parse(body).answers[0].data;
                   columns = type[0].columns;
                   key = Object.keys(columns);
                   count = JSON.parse(body).answers[0].metadata.count;
                   console.log(key);
                   for (i = 0; i < count; i++) {
                       answer = "";
                       answer =key[0].toUpperCase() + ": " + data[i][key[0]] + "\n" + key[1].toUpperCase() + ": " + data[i][key[1]] + "\n" + key[2].toUpperCase() + ": " + data[i][key[2]] + "\n\n";
                       bot.sendMessage(from, answer);
                   }
               } else if (type.length == 2 && type[1].type == "rss"){
                 data = (JSON.parse(body)).answers[0].data;
                 columns = type[1];
                 key = Object.keys(columns);
                 for(i = 0; i< 4; i++){
                     if(i == 0){
                         answer = (JSON.parse(body)).answers[0].actions[0].expression;
                         bot.sendMessage(from, answer);
                     } else {
                         answer = "";
                         answer = key[1].toUpperCase() + ": " + data[i][key[1]] + "\n" + key[2].toUpperCase() + ": " + data[i][key[2]] + "\n" + key[3].toUpperCase() + ": " + data[i][key[3]] + "\n\n";
                         bot.sendMessage(from, answer);
                     }
                 }
               } else {
                   answer = "Oops looks like SUSI is taking break try to contact later.";
                   bot.sendMessage(from, answer);
               }
       });
    });
    
    app.listen(app.get('port'));
    
  8. Enter your email in above code.Now, we have to deploy the above-written code onto Heroku, but first, we will make a Github repository and push the files to Github as we will use Github to deploy our code.
    In command line change current directory to folder we created above and  write

    git init
    git add .
    git commit -m”initial”
    git remote add origin <URL for remote repository>
    git remote -v
    git push -u origin master

    You will get URL for the remote repository by making a repository on your Github and copying this link of your repository.

  9. Now we have to deploy this Github repository to Heroku to get URL.
  10. If you don’t have an account on Heroku sign up here https://www.heroku.com/ else just sign in and create a new app.
  11. Deploy your repository onto Heroku from deploy option and choosing Github as a deployment method.
  12. Select automatic deployment so that when you make any changes in the Github repository, they are automatically deployed to heroku.

You have successfully made SUSI Hangout bot. Try massaging it from your other account on https://hangouts.google.com

Resources

Hangouts bot in python: https://github.com/hangoutsbot/hangoutsbot

Continue ReadingHow to Make SUSI Bot for Google Hangouts

How to Deploy Node js App to Google Container Engine Using Kubernetes

There are many ways to host node js apps. A popular way is to host your app on Heroku. We can also deploy our app to Google cloud platform on container engine using Kubernetes. In this blog, we will learn how to deploy node js app on container engine with kubernetes. There are many resources on the web but we will use YAML files to create a deployment and to build docker image we will not use Google container registry (GCR) as it will cost us more for the deployment. To deploy we will start by creating an account on Google cloud and you can get a free tier of Google cloud platform worth 300$ for 12 months. After creating account create a project with any name of your choice. Enable Google cloud shell from an icon on right top.

We will also need a docker image of our app for deployment. To create a docker image first create an account on https://www.docker.com and create a repository with any name on it. Now, we will add docker file into our repository so that we can build docker image. Dockerfile will contain this code:

FROM node:boron
# Create app directory
RUN mkdir -p /usr/src/app
WORKDIR /usr/src/app
# Install app dependencies
COPY package.json /usr/src/app
RUN npm install
# Bundle app source
COPY . /usr/src/app
EXPOSE 8080
CMD [ "npm", "start" ]

You can see an example of docker file in SUSI telegram repository. After pushing docker file to your repository we will now build docker image of the app. In Google cloud shell clone, your repository with git clone {your-repository-link here }and change your current directory to the cloned app. We will use –no-cache -t arguments as described above it will be better for building docker image and it will use fewer resources. Run these two commands to build docker image and pushing it to your docker hub.

docker build --no-cache -t {your-docker-username-here}/{repository-name-on-docker here} .

docker push {your-docker-username-here}/{repository-name-on-docker here}


We have successfully created docker image for our app. Now we will deploy our app to container engine using this image. To deploy it we will use configuration files. Add a yaml folder in your repository and we will add four files into it now. In the first file, we will specify the namespace and name it as 00-namespace.yml It will contain following code:

apiVersion: v1
kind: Namespace
metadata:
 name: web

In second file we will configure our namespace that we specified and name it as configmap.yml It will contain following code:

apiVersion: v1
metadata:
 name:{name-of-your-deployment-here}
 namespace: web
kind: ConfigMap

In third file, we will define our deployment and name it as deployment.yml It will contain following code:

kind: Deployment
apiVersion: apps/v1beta1
metadata:
 name: {name-of-your-deployment-here}
 namespace: web
spec:
 replicas: 1
 template:
   metadata:
     labels:
       app: {name-of-your-deployment-here}
   spec:
     containers:
     - name: {name-of-your-deployment-here}
       image: {your-docker-username-here}/{repository-name-on-docker here}:latest
       ports:
       - containerPort: 8080
         protocol: TCP
       envFrom:
       - configMapRef:
           name: {name-of-your-deployment-here}
     restartPolicy: Always

In fourth file, we will define service for our deployment and name it as service.yml It will contain following code:

kind: Service
apiVersion: v1
metadata:
 name: {name-of-your-deployment-here}
 namespace: web
spec:
 ports:
 - port: 8080
   protocol: TCP
   targetPort: 8080
 selector:
   app: {name-of-your-deployment-here}
 type: LoadBalancer

After adding these files to repository we will now deploy our app to a cluster on container engine. Go to Google cloud shell and run the following commands:

gcloud config set compute/zone us-central1-b

This will set zone for our cluster.

gcloud container clusters create {name-your-cluster-here}

Now update in your repository with git pull as we have added new files to it. Run the following command to make your deployment and to see it:

kubectl create -R -f ./yamls

This will create our deployment.

kubectl get services --namespace=web


With above command, you will get external IP for your app and open that IP in your browser with the port. You will see your app.

kubectl get deployments --namespace=web

Run this command if available is 1 then it means your deployment is running the file.

You have successfully deployed your app to container engine using kubernetes.

Resources

Tutorial on Google cloud: https://cloud.google.com/container-engine/docs/tutorials/hello-node
Tutorial by Jatin Shridhar: https://www.sitepoint.com/kubernetes-deploy-node-js-docker-app/

Continue ReadingHow to Deploy Node js App to Google Container Engine Using Kubernetes

Adding Servlet for SUSI Service

Whenever we ask anything to SUSI, we get a intelligent reply. The API endpoints which clients uses to give the reply to users is http://api.susi.ai/susi/chat.json, and this API endpoint is defined in SUSIService.java. In this blog post I will explain how any query is processed by SUSI Server and the output is sent to clients.

public class SusiService extends AbstractAPIHandler implements APIHandler

This is a public class and just like every other servlet in SUSI Server, this extends AbstractAPIHandler class, which provides us many options including AAA related features.

    public String getAPIPath() {
           return "/susi/chat.json";
    }

The function above defines the endpoint of the servlet. In this case it is  “/susi/chat.json”. The final API Endpoint will be API http://api.susi.ai/susi/chat.json.

This endpoint accepts 6 parameters in GET request . “q” is the query “parameter”, “timezoneOffset” and “language” parameters are for giving user a reply according to his local time and local language, “latitude” and “longitude” are used for getting user’s location.

      String q = post.get("q", "").trim();
        int count = post.get("count", 1);
        int timezoneOffset = post.get("timezoneOffset", 0); // minutes, i.e. -60
        double latitude = post.get("latitude", Double.NaN); // i.e. 8.68
        double longitude = post.get("longitude", Double.NaN); // i.e. 50.11
        String language = post.get("language", "en"); // ISO 639-1

After getting all the parameters we do a database update of the skill data. This is done using DAO.susi.observe() function.

Then SUSI checks that whether the reply was to be given from a etherpad dream, so we check if we are dreaming something.

if (etherpad_dream != null && etherpad_dream.length() != 0) {
    String padurl = etherpadUrlstub + "/api/1/getText?apikey=" + etherpadApikey + "&padID=$query$";

If SUSI is dreaming something then we call the etherpad API with the API key and padID.

After we get the response from the etherpad API we parse the JSON and store the text from the skill in a local variable.

JSONObject json = new JSONObject(serviceResponse);
String text = json.getJSONObject("data").getString("text");

After that, we fill an empty SUSI mind with the dream. SUSI susi_memory_dir is a folder named as “susi” in the data folder, present at the server itself.

SusiMind dream = new SusiMind(DAO.susi_memory_dir); 

We need the memory directory here to get a share on the memory of previous dialogues, otherwise, we cannot test call-back questions.

JSONObject rules = SusiSkill.readEzDSkill(new BufferedReader(new InputStreamReader(new ByteArrayInputStream(text.getBytes(StandardCharsets.UTF_8)), StandardCharsets.UTF_8)));
dream.learn(rules, new File("file://" + etherpad_dream));

When we call dream.learn function, Susi starts dreaming. Now we will try to find an answer out of the dream.

The SusiCognition takes all the parameters including the dream name, query and user’s identity and then try to find the answer to that query based on the skill it just learnt.

    SusiCognition cognition = new SusiCognition(dream, q, timezoneOffset, latitude, longitude, language, count, user.getIdentity());

If SUSI finds an answer then it replies back with that answers else it tries to answer with built-in intents. These intents are both existing SUSI Skills and internal console services.

if (cognition.getAnswers().size() > 0) {          DAO.susi.getMemories().addCognition(user.getIdentity().getClient(), cognition);
    return new ServiceResponse(cognition.getJSON());
}

This is how any query is processed by SUSI Server and the output is sent to clients.

Currently people use Etherpad (http://dream.susi.ai/) to make their own skills, but we are also working on SUSI CMS where we can edit and test the skills on the same web application.

References

Java Servlet Docs: http://docs.oracle.com/javaee/6/tutorial/doc/bnafd.html

Embedding Jetty: http://www.eclipse.org/jetty/documentation/9.4.x/embedding-jetty.html

Server-Side Java: http://www.javaworld.com/article/2077634/java-web-development/how-to-get-started-with-server-side-java.html

Continue ReadingAdding Servlet for SUSI Service

Skill Editor in SUSI Skill CMS

SUSI Skill CMS is a web application built on ReactJS framework for creating and editing SUSI skills easily. It follows an API centric approach where the SUSI Server acts as an API server. In this blogpost we will see how to add a component which can be used to create a new skill SUSI Skill CMS.

For creating any skill in SUSI we need four parameters i.e model, group, language, skill name. So we need to ask these 4 parameters from the user. For input purposes we have a common card component which has dropdowns for selecting models, groups and languages, and a text field for skill name input.

<SelectField
    floatingLabelText="Model"
    value={this.state.modelValue}
    onChange={this.handleModelChange}
>
    {models}
</SelectField>
<SelectField
    floatingLabelText="Group"
    value={this.state.groupValue}
    onChange={this.handleGroupChange}
>
    {groups}
</SelectField>
<SelectField
    floatingLabelText="Language"
    value={this.state.languageValue}
    onChange={this.handleLanguageChange}
>
    {languages}
</SelectField>
<TextField
    floatingLabelText="Enter Skill name"
    floatingLabelFixed={true}
    hintText="My SUSI Skill"
    onChange={this.handleExpertChange}
/>
<RaisedButton label="Save" backgroundColor="#4285f4" labelColor="#fff" style={{marginLeft:10}} onTouchTap={this.saveClick} />

This is the card component where we get the user input. We have API endpoints on SUSI Server for getting the list of models, groups and languages. Using those APIs we inflate the dropdowns.
Then the user needs to edit the skill. For editing of skills we have used Ace Editor. Ace is an code
editor written in pure javascript. It matches the features native editors like Sublime and TextMate.

To use Ace we need to install the component.

npm install react-ace --save                        

This command will install the dependency and update the package.json file in our project with this dependency.

To use this editor we need to import AceEditor and place it in the render function of our react class.

<AceEditor
    mode=" markup"
    theme={this.state.editorTheme}
    width="100%"
    fontSize={this.state.fontSizeCode}
    height= "400px"
    value={this.state.code}
    name="skill_code_editor"
    onChange={this.onChange}
    editorProps={{$blockScrolling: true}}
/>

Now we have a page that looks something like this

Now we need to handle the click event when a user clicks on the save button.

First we check if the user is logged in or not. For this we check if we have the required cookies and the access token of the user.

 if(!cookies.get('loggedIn')) {
            notification.open({
                message: 'Not logged In',
                description: 'Please login and then try to create/edit a skill',
                icon: <Icon type="close-circle" style={{ color: '#f44336' }} />,
            });
            return 0;
        }

If the user is not logged in then we show him a error notification and asks him to login.

Then we check if he has filled all the required fields like name of the skill etc. and after that we call an API Endpoint on SUSI Server that will finally store the skill in the skill_data_repo.

let url= “http://api.susi.ai/cms/modifySkill.json”
$.ajax({
    url:url,
    dataType: 'jsonp',
    jsonp: 'callback',
    crossDomain: true,
    success: function (data) {
        console.log(data);
        if(data.accepted===true){
            notification.open({
                message: 'Accepted',
                description: 'Your Skill has been uploaded to the server',
                icon: <Icon type="check-circle" style={{ color: '#00C853' }} />,
            });
           }
    }
});

In the success function of ajax call we check if accepted parameter is true from the server or not. If accepted is true then we show user a notification with a message that “Your Skill has been uploaded to the server”.

To see this component running please visit http://skills.susi.ai/skillEditor.

Resources

Material-UI: http://www.material-ui.com/

Ace Editor: https://github.com/securingsincity/react-ace

Ajax: http://api.jquery.com/jquery.ajax/

Universal Cookies: https://www.npmjs.com/package/universal-cookie

Continue ReadingSkill Editor in SUSI Skill CMS

Uploading Images to SUSI Server

SUSI Skill CMS is a web app to create and modify SUSI Skills. It needs API Endpoints to function and SUSI Server makes it possible. In this blogpost, we will see how to add a servlet to SUSI Server to upload images and files.

The CreateSkillService.java file is the servlet which handles the process of creating new Skills. It requires different user roles to be implemented and hence it extends the AbstractAPIHandler.

Image upload is only possible via a POST request so we will first override the doPost method in this servlet.

  @Override
  protected void doPost(HttpServletRequest req, HttpServletResponse resp) throws ServletException, IOException {
  resp.setHeader("Access-Control-Allow-Origin", "*"); // enable CORS

resp.setHeader enables the CORS for the servlet. This is required as POST requests must have CORS enables from the server. This is an important security feature that is provided by the browser.

        Part file = req.getPart("image");
        if (file == null) {
            json.put("accepted", false);
            json.put("message", "Image not given");
        }

Image upload to servers is usually a Multipart Request. So we get the part which is named as “image” in the form data.

When we receive the image file, then we check if the image with the same name exists on the server or not.

Path p = Paths.get(language + File.separator + “images/” + image_name);

        if (image_name == null || Files.exists(p)) {
                json.put("accepted", false);
                json.put("message", "The Image name not given or Image with same name is already present ");
            }

If the same file is present on the server then we return an error to the user requesting to give a unique filename to upload.

Image image = ImageIO.read(filecontent);
BufferedImage bi = this.createResizedCopy(image, 512, 512, true);
if(!Files.exists(Paths.get(language.getPath() + File.separator + "images"))){
   new File(language.getPath() + File.separator + "images").mkdirs();
           }
ImageIO.write(bi, "jpg", new File(language.getPath() + File.separator + "images/" + image_name));

Then we read the content for the image in an Image object. Then we check if images directory exists or not. If there is no image directory in the skill path specified then create a folder named “images”.

We usually prefer square images at the Skill CMS. So we create a resized copy of the image of 512×512 dimensions and save that copy to the directory we created above.

BufferedImage createResizedCopy(Image originalImage, int scaledWidth, int scaledHeight, boolean preserveAlpha) {
        int imageType = preserveAlpha ? BufferedImage.TYPE_INT_RGB : BufferedImage.TYPE_INT_ARGB;
        BufferedImage scaledBI = new BufferedImage(scaledWidth, scaledHeight, imageType);
        Graphics2D g = scaledBI.createGraphics();
        if (preserveAlpha) {
            g.setComposite(AlphaComposite.Src);
        }
        g.drawImage(originalImage, 0, 0, scaledWidth, scaledHeight, null);
        g.dispose();
        return scaledBI;
    }

The function above is used to create a  resized copy of the image of specified dimensions. If the image was a PNG then it also preserves the transparency of the image while creating a copy.

Since the SUSI server follows an API centric approach, all servlets respond in JSON.

       resp.setContentType("application/json");
       resp.setCharacterEncoding("UTF-8");
       resp.getWriter().write(json.toString());’

At last, we set the character encoding and the character set of the output. This helps the clients to parse the data easily.

To see this endpoint in live send a POST request at http://api.susi.ai/cms/createSkill.json.

Resources

Apache Docs: https://commons.apache.org/proper/commons-fileupload/using.html

Multipart POST Request Tutorial: http://www.codejava.net/java-se/networking/upload-files-by-sending-multipart-request-programmatically

Java File Upload tutorial: https://ursaj.com/upload-files-in-java-with-servlet-api

Jetty Project: https://github.com/jetty-project/

Continue ReadingUploading Images to SUSI Server

Adding Map and RSS Action Type Support to SUSI MagicMirror Module with React

SUSI being an interactive personal assistant, answers questions in a variety of formats. This includes maps, RSS, table, and pie-chart. SUSI MagicMirror Module earlier provided support for only Answer Action Type. So, if you were to ask about a location, it could not show you a map for that location. Support for a variety of formats was added to SUSI Module for MagicMirror so that users can benefit from rich responses by SUSI.AI.

One problem that was faced while adding UI components is that in the MagicMirror Module structure, each module needs to supply its DOM by overriding the getDom() method. Therefore, you need to manage all the UI programmatically. Managing UI programmatically in Javascript is a cumbersome task since you need to create DOM nodes, manually apply styling to them, and add them to parent DOM object which is needed to be returned. We need to write UI for each element like below:

getDom: function () {
        .... 
        ....
        const moduleDiv = document.createElement("div");

        const visualizerCanvas = document.createElement("canvas");
        moduleDiv.appendChild(visualizerCanvas);

        const mapDiv = document.createElement("div");
        loadMap(mapDiv,lat, long);
        moduleDiv.appendChild(mapDiv);
        ...
        ...
}

As you can see, manually managing the DOM is neither that easy nor a recommended practice. It can be done in a more efficient way using the React Library by Facebook.  React is an open source UI library by Facebook. It works on the concept of Virtual DOM i.e. the whole DOM object gets created in the memory and only the changed components are reflected on the document.

Since the SUSI MagicMirror Module is primarily written in open-source TypeScript Lang (a typed superset of JavaScript), we also need to write React in TypeScript. To add React to a Typescript Project, we need to add some dependencies. They can be added using:

$ yarn add react react-dom @types/react @types/react

Now, we need to change our Webpack config to build .tsx files for React. TSX like JSX can contain HTML like syntax for representing DOM object in a syntactic sugar form. This can be done by changing resolve extensions and loaders config so that awesome typescript loaded compiles that TSX files. It is needed to be modified like below

resolve: {
   extensions: [".js", ".ts", ".tsx", ".jsx"],
},

module: {
   loaders: [{
       test: /\.tsx?$/,
       loaders: ["awesome-typescript-loader"],
   },
       {
           test: /\.json$/,
           loaders: ["json-loader"],
       }],
},

This will allow webpack to build and load both .tsx and .ts files. Now that project is setup properly, we need to add UI for Map and RSS Action Type.

The UI for Map is added with the help of React-Leaflet library. React-Leaflet module is a module build on top of Leaflet Map library for loading maps in Browser. We add the React-Leaflet library using

$ yarn add react-leaflet

Now, we declare a MapView Component in React and render Map in it using the React-Leaflet Library. Custom styling can be applied to it. The render function for MapView React Component is defined as follows.

import * as React from "react";
import {Map, Marker, Popup, TileLayer} from "react-leaflet";
interface IMapProps {
   latitude: number;
   longitude: number;
   zoom: number;
}

export class MapView extends React.Component<IMapProps, any> {

   public constructor(props: IMapProps) {
       super(props);
   }

   public render(): JSX.Element | any | any {
       const center = [this.props.latitude, this.props.longitude];
       console.log(center);
       return <Map center={center} zoom={this.props.zoom} style={{height: "300px"}}>
           <TileLayer url="http://{s}.tile.osm.org/{z}/{x}/{y}.png"/>
           <Marker position={center}>
               <Popup>
                   <span> Here </span>
               </Popup>
           </Marker>
       </Map>;
   }
}

For making the UI for RSS Action Type, we define an RSS Card Component. An RSS feed is constituted by various RSS Cards. An RSS Card is defined as follows.

import * as React from "react";

export interface IRssProps {
   title: string;
   description: string;
   link: string;
}

export class RSSCard extends React.Component <IRssProps, any> {

   constructor(props: IRssProps) {
       super(props);
   }

   public render(): JSX.Element | any | any {
       return <div className="card">
           <div className="card-title">{this.props.title}</div>
           <div className="card-description">{this.props.description}</div>
       </div>;
   }
}

Now, we define an RSS feed which is constituted by various RSS Information Cards. Since screen size is limited and there is no option available to the user to scroll, we limit the number of cards displayed to 5 with slice operation on data array.

import * as React from "react";
import {IRssProps, RSSCard} from "./rss-card";

export interface IRSSFeedProps {
   feeds: Array<IRssProps>;
}

export class RSSFeed extends React.Component <IRSSFeedProps, any> {

   public constructor(props: IRSSFeedProps) {
       super(props);
   }

   public render(): JSX.Element | any | any {
       return <div className="rss-div">
           {this.props.feeds.map((feed: IRssProps) => {
                   return <RSSCard key={feed.title} title={feed.title} description={feed.description} link={feed.link}/>;
               }
           ).slice(0, 5)}
       </div>;
   }
}

Now, we can add these components to UI easily and render it with ReactDOM like:

ReactDOM.render(<TableView data={tableData} columns={action.columns}/>, tableDiv);

Below is an example screenshot of RSS and Map View in SUSI MagicMirror.

Resources:

Continue ReadingAdding Map and RSS Action Type Support to SUSI MagicMirror Module with React

Adding Face Recognition based Authentication to SUSI MagicMirror Module

SUSI MagicMirror Module is a module designed for MagicMirror that helps you get SUSI Intelligence right on your Mirror. You may then ask it questions in the way the Queen in the tale “Snow White and the Seven Dwarfs” asked. One key feature that was missing in it was that the user could be recognized and queries he asked are answered in a personalized manner. This could be achieved if SUSI uses the account dedicated to that person to answer his/her queries. Thus, we need an authentication support.

The authentication on MagicMirror is not as trivial as on Web, Android and iOS client apps for SUSI. Key difference here is that user, while using the MagicMirror, does not have access to a keyboard and mouse. Therefore, we cannot simply ask him to input email and password. Furthermore, a MagicMirror installed in your home may be used by several members of your family. Thus, we need a mechanism to tell each user apart.

This was done with the help of MMM-Facial-Recognition module which brings face recognition support to MagicMirror.

MMM-Facial-Recognition module provides support for recognizing multiple faces and setting the modules on the mirror screen based on the user facing the mirror using OpenCV. Other modules can also take advantage of knowing about the person with the help of module notifications sent by MMM-Facial-Recognition Module.

To add Face based Authentication support to SUSI with MMM-Facial-Recognition, we first need to add the latter to MagicMirror. It can be added easily by first cloning the repository to modules directory of MagicMirror.

$ git clone https://github.com/paviro/MMM-Facial-Recognition

Go inside the directory and install dependencies

$ npm run install

Now, we need to train a model for the users who are going to use the MagicMirror. This can be done by the MMM-Facial-Recognition-Tools. This tool captures photos from the camera and trains a model for Face Recognition. The guide to use the tool is very well written on the Github page so I am not including it here. After training for faces of the users, you will get a training.xml file. This file contains the information about the facial features of every person so that it can tell users apart. You need to copy this file to the Module directory for MMM-Facial-Recognition module i.e. MagicMirror/module/MMM-Facial-Recognition.

After this we can add the module to MagicMirror, by modifying the config file. Add the following lines in the config file (config.js). Copy and paster username array from the training script in the asked position.

{
    module: 'MMM-Facial-Recognition',
    config: {
        // 1=LBPH | 2=Fisher | 3=Eigen
        recognitionAlgorithm: 1,
        lbphThreshold: 50,
        fisherThreshold: 250,
        eigenThreshold: 3000,
        useUSBCam: true,
        trainingFile: 'modules/MMM-Facial-Recognition/training.xml',
        interval: 2,
        logoutDelay: 15,
        // Array with usernames (copy and paste from training script)
        users: [],
        defaultClass: "default",
        everyoneClass: "everyone",
        welcomeMessage: true
    }
}

You may configure the show and hide behavior of modules based on the person. Find more information about it in the official guide on the repository. After setting up it recognizes and shows welcome message to each user like this.

 

Now, we need to integrate this module to SUSI for Authentication. To do this first of all we make config for SUSI MagicMirror Module to add user authentication along with their name registered on Facial Recognition Module. It can be done by adding SUSI MagicMirror module config file (config.js) like below.

{
       module: "MMM-SUSI-AI",
       position: "top_center",
       config: {
            hotword: "Susi",
            users: [{
                face_recognition_username: "Pranjal Paliwal",
                email: "paliwal.pranjal83@gmail.com",
                password: "PASSWORD_HERE"
            }, {
                face_recognition_username: "Chashmeet Singh",
                email: "chashmeetsingh@gmail.com",
                password: "PASSWORD_HERE"
            }],
        },
        classes: 'default everyone'
},

Now, we need to know that which user is facing the mirror at that time. MMM-Facial-Recognition sends a module notification when a user is detected. The format of the notification is

sender : MMM-Facial-Recognition
type: CURRENT_USER
payload: Name of the User / None 

If the user is recognized we get the name of the User as payload. If no face could be identified, we get None as payload.

We need to find out user based on the user’s name registered in the module. We already have that parameter in the user object in users array in config for SUSI MagicMirror Module (MMM-SUSI-AI). We can iterate over users array to find out the user facing the mirror on receiving the notification. In SUSI Chat API, users are identified with the help of an access token. On identifying a user, we perform login with the help of SignInService to obtain token for him. The implementation of the above task can be understood via the following snippet.

public receivedNotification(type: NotificationType, payload: any): void {
   if (type === "CURRENT_USER") {
       console.log("Current User", payload);
       if (payload === "None") {
           this.configService.Config.accessToken = null;
       } else {
           console.log(this.config.users);
           for (const user of this.config.users) {
               if (user.face_recognition_username === payload) {
                   if (isUndefined(this.signInService)) {
                       this.signInService = new SignInService(user);
                   }
                   this.signInService.updateUser(user).then((token) => {
                       console.log("updating token for " + user);
                       this.configService.Config.accessToken = token;
                   });
                   return;
               }
           }
           this.configService.Config.accessToken = null;
       }
   }
}

Explanation: In the receivedNotification method of the Main Component of SUSI MagicMirror module, we check if notification is of type CURRENT_USER. If the payload is None, we set access-token to null. If a user is identified, we check if it is contained in the users array. If present, we perform Sign In to SUSI Server for that user and store the access token obtained in the Config.

Now, every time a recognized my Facial Recognition module, the access token is updated in the config. We use the accessToken field in Config to send the message to SUSI Chat API. The implementation of it can be referred below.

public async askSusi(query: string): Promise<any> {

   const accessToken = this.configService.Config.accessToken;

   const requestString: string = (!isUndefined(accessToken) && accessToken != null) ?
       `http://api.susi.ai/susi/chat.json?q=${query}&access_token=${accessToken}` :
       `http://api.susi.ai/susi/chat.json?q=${query}`;

   const response = await WebRequest.get(requestString);
   return JSON.parse(response.content);
}

By using the above approach, the request sent to SUSI Server are identified according to the person facing the mirror. SUSI can, therefore, answer according to the user. In this way, authentication with Face Recognition is performed in the SUSI Magic Mirror Module.

Resources

 

Continue ReadingAdding Face Recognition based Authentication to SUSI MagicMirror Module

Enhancing SUSI Line Bot UI by Showing Carousels and Location Map

A good UI primarily focuses on attracting large numbers of users and any good UI designer aims to achieve user satisfaction with a user-friendly design. In this blog, we will learn how to show carousels and location map in SUSI line bot to make UI better and easy to use. We can show web search result from SUSI in form of text responses in line bot but it doesn’t follow design rule as it is not a good approach to show more text in constricted space. Users deem such a layout as less attractive. In SUSI webchat, location is returned with a map which looks more promising to users as illustrated below:

Web search is returned as carousels and is viewable as:

We can implement web search in line by sending an array of text responses. We can do this with the code below:

for (var i = 0; i < 5; i++) {
   msg[i] = "";
   msg[i] = {
       type: 'text',
       text: 'text to be sent here'
   }
}
return client.replyMessage(event.replyToken, msg);

If we send web search using text response it looks messy, a user will not like it that much as it is difficult for a user to read and understand a lot of text in one place. You can see it in the screenshot below:

To make it better looking we will implement carousels for web search in SUSI line bot. Carousels are tiles that can be scrolled horizontally to show rich content. We can implement carousels using this code:

for (var i = 1; i < 4; i++) {
   title = 'title of carousel';
   query = title;
   msg = 'description of carousel here';
   link = 'link to be opened here';
   if (title.length >= 40) {
       title = title.substring(0, 36);
       title = title + "...";
   }
   if (msg.length >= 60) {
       msg = msg.substring(0, 56);
       msg = msg + "...";
   }
   carousel[i] = {
       "title": title,
       "text": msg,
       "actions": [{
               "type": "uri",
               "label": "View detail",
               "uri": link
           },
           {
               "type": "message",
               "label": "Ask SUSI again",
               "text": query
           }
       ]
   };
}

In above code, we are checking the length of title and description because line API has a limit for the title up to 40 characters and description up to 60 characters. If limit exceeds we then trim the title or description accordingly and use it. Next, we assign title, description, and link to be opened in action to carousel so that we can use it in below code.

const answer = [{
       type: 'text',
       text: ans
   },
   {
       "type": "template",
       "altText": "this is a carousel template",
       "template": {
           "type": "carousel",
           "columns": [
               carousel[1],
               carousel[2],
               carousel[3]
           ]
       }
   }
]
return client.replyMessage(event.replyToken, answer);

Above code shows an array names answer in which we have added carousels that we have created in last code. After implementing this web search will look like this:

This UI looks neat and easy to read and users will like this. Ask SUSI again will send the title of the carousel to SUSI. To show location map on SUSI line bot just like it is shown on SUSI webchat we will follow this code:

const answer = [{
       type: 'text',
       text: ans
   },
   {
       "type": "location",
       "title": "name of the place here",
       "address": address,
       "latitude": latitude of location here,
       "longitude": longitude of location here
   }
]
return client.replyMessage(event.replyToken, answer);

We will send a location type message which will include latitude and longitude of the location so that Line bot can show location using map. After implementing location type response it will look like this:

On clicking on this location it will open a map inside line app on which we will see a pointer pointing at the location that we have provided.

If you want to learn more about line messaging API refer to this https://devdocs.line.me/en/#messaging-api

Resources
https://devdocs.line.me/en/#messaging-api

Continue ReadingEnhancing SUSI Line Bot UI by Showing Carousels and Location Map