Implementing Real Time Speech to Text in SUSI Android App

Android has a very interesting feature built in called Speech to Text. As the name suggests, it converts user’s speech into text. So, android provides this feature in form of a recognizer intent. Just the below 4 line code is needed to start recognizer intent to capture speech and convert into text. This was implemented earlier in SUSI Android App too.  Simple, isn’t it?

Intent intent = new Intent(RecognizerIntent.ACTION_RECOGNIZE_SPEECH);
intent.putExtra(RecognizerIntent.EXTRA_LANGUAGE_MODEL,
       RecognizerIntent.LANGUAGE_MODEL_FREE_FORM);
intent.putExtra(RecognizerIntent.EXTRA_CALLING_PACKAGE,
       "com.domain.app");
startActivityForResult(intent, REQ_CODE_SPEECH_INPUT);

But there is a problem here. It popups an alert dialog box to take a speech input like this :

Although this is great and working but there were many disadvantages of using this :

  1. It breaks connection between user and app.
  2. When user speaks, it first listens to complete sentence and then convert it into text. So, user was not able to see what the first word of sentence was until he says the complete sentence.
  3. There is no way for user to check if the sentence he is speaking is actually converting correctly to speech or not. He only gets to know about it when the sentence is completed.
  4. User can not cancel the conversion of Speech to text if some word is converted wrongly because the user doesn’t even get to know if the word is converted wrongly. It is kind of suspense that a whether the conversion is right or not.

So, now the main challenge was how can we improve the Speech to Text and implement it something like Google now?  So, what google now does is, it converts speech to text alongside taking input making it a real time speech to text convertor. So, basically if you speak second word of sentence, the first word is already converted and displayed. You can also cancel the conversion of rest of the sentence if the first word of sentence is converted wrongly.

I searched this problem a lot on internet, read various blogs and stackoverflow answers, read Documentation of Speech to Text until I finally came across SpeechRecognizer class and RecognitionListener. The plan was to use object of SpeechRecognizer class with RecognizerIntent and RecognitionListener to generate partial results.

So, I just added this line to the the above code snippet of recognizer intent.

intent.putExtra(RecognizerIntent.EXTRA_PARTIAL_RESULTS,true);

And introduced SpeechRecognizer class and RecognitionListener like this

 SpeechRecognizer recognizer = SpeechRecognizer
   .createSpeechRecognizer(this.getApplicationContext());

RecognitionListener listener = new RecognitionListener() {
   @Override
   public void onResults(Bundle results) {
   }
 
   @Override
   public void onError(int error) {
   }
 
   @Override
   public void onPartialResults(Bundle partialResults) {
    //The main method required. Just keep printing results in this.
   }
};

Now combine SpeechRecognizer class with RecognizerIntent and RecognitionListener with these two lines.

recognizer.setRecognitionListener(listener);
recognizer.startListening(intent);

Everything is now set up. You can now use result in onPartialResults method and keep displaying partial results. So, if you want to speak “My name is…” , “My” will be displayed on screen by the time you speak “name” and “My name” will be displayed on screen by the time you speak “is” . Isn’t it cool?

This is how Realtime Speech to Text is implemented in SUSI Android App. If you want to implement this in your app too, just follow the above steps and you are good to go. So, the results look like these.

  

Resources

Continue ReadingImplementing Real Time Speech to Text in SUSI Android App

Tray Icon for SUSI Desktop Client

Susi is an AI which replies smartly to anything that you put. Popular Assistants like Siri and Cortana, have their own tray icons that run on top of all apps. So, I thought to add a tray icon to SUSI AI as well.

A tray is an icon in an OS’s notification area usually attached with a context menu.

Platform limitations:

  • On Linux, app indicator is used if it is supported else GtkStatusIcon is used
  • App indicator is only shown when it has a context menu.
  • When app indicator is used all click events are ignored.

Starting Up:

You should have Node and ElectronJS installed

To create a tray I made two methods. One is for creating a tray and other is for creating a window when a tray icon is clicked.

app.on('ready', () => {
 createTray()
 createWindow()
 })

app.on(‘ready’,() => {}) is called when the electron is ready to run. It’s just like the main function.

Then we create the Tray.

const createTray = () => {
 tray = new Tray(path.join(__dirname, 'static/icons/png/tray.png'))
 tray.on('right-click', toggleWindow)
 tray.on('double-click', toggleWindow)
 tray.on('click', function (event) {
 toggleWindow()
 

When some developer is using SUSI he might // Show devtools when command clicked if 

(window.isVisible() && process.defaultApp && event.metaKey) {
 window.openDevTools({mode: 'detach'}) }}) }
Tray = new Tray(ICON HERE) // this creates a new tray with the icon specified.

Since there are 2 themes for the notification bar in OSX, I used a BLUE icon that looks good on both the themes.

I Used the icon above as a tray icon for SUSI.

Then next lines are for opening the tray with right-click or double-click. We can also change these functionalities like on right-click we can open some settings or give user an option to close the app.

const createWindow = () => {
  window = new BrowserWindow({
    width: 300,
    height: 450,
    show: false,
    frame: false,
    fullscreenable: false,
    resizable: false,
    transparent: true,
    webPreferences: {
      // Prevents renderer process code from not running when window is
      // hidden
      backgroundThrottling: false
    }
  })
  window.loadURL(`file://${path.join(__dirname, '/static/index.html')}`)
  // show inspect element in the window
  // window.openDevTools();
  // set window to always be on the top of other windows
  window.setAlwaysOnTop(true);
  // Hide the window when it loses focus
  window.on('blur', () => {
    if (!window.webContents.isDevToolsOpened()) {
      window.hide()
    }
  })
}

 

When we call createWindow it creates a Window with the HTML File specified in loadURL

window.loadURL(`file://${path.join(__dirname, '/static/index.html')}`)

In this we created a SUSI Chat Client in HTML and named it as index.html and passed that HTML to be loaded as the application window.

Sometimes when we a running a full screen, Especially in macOS then all apps are launched behind the fullscreen app. To run our tray icon we used another function that set the window to be always on top of all other windows.

 window.setAlwaysOnTop(true);

So if someone is reading a news article and wants to find a meaning of a word he need not open SUSI again, because SUSI is running on top of all the applications.

toggleWindow is called when we click the Tray Icon. The work of this function is fairly simple. On click of the tray icon, it makes hides or shows the SUSI application window.

const toggleWindow = () => {
  if (window.isVisible()) {
    window.hide()
  } else {
    showWindow()
  }
}

showWindow is the function that shows the windows and brings it to focus.

This function brings the SUSI Window to focus and shows it to the user.

const showWindow = () => {
 const position = getWindowPosition()
 window.setPosition(position.x, position.y, false)
 window.show()
 window.focus()
}

Running

Navigate to the project directory and run:

electron . 

SUSI icon will be visible in your notification tray.

Resources:

SUSI Desktop App: https://github.com/fossasia/susi_desktop/ 

Electron Documentation: https://electron.atom.io/

Electron Packager: https://github.com/electron-userland/electron-packager

Continue ReadingTray Icon for SUSI Desktop Client

SUSI Desktop App in Electron

 

Electron allows you to get a cross-platform desktop application up and running very quickly. An electron app is built using the node.js framework and is brought to us by Github. Apps built with Electron are just web sites which are opened in an embedded Chromium web browser.

Setting Up.

First things first, we need NodeJS installed. So go to https://nodejs.org/en/download/ and follow the instructions as per your Operating System.

Now we need Electron. We can install it globally and use it as a CLI, or install it locally in our application’s path. I recommend installing it globally, so that every app does not need to install it every time.

Open terminal and write:

$ npm install -g electron-prebuilt
$ npm install -g bower


Bower is used for managing front end components like HTML, CSS, JS etc.

To test the installation, type electron -h and it should show the version of Electron CLI.

Getting the project ready.

The main idea behind making desktop apps with JS is that you build one codebase and package it for each operating system separately. This abstracts away the knowledge needed to make native desktop apps and makes maintenance a lot easier.

First clone the SUSI Desktop app repository on your computer.

$ git clone https://github.com/fossasia/susi_desktop.git

Next step is to change directory to susi_desktop.

$ cd susi_desktop

Now we need to install dependencies which are used by our app.

$ npm install
$ bower install

Now we have the code and the dependencies setup. To run the Application write

$ npm start

Or

$ electron .

Any of the two commands can be used to run a electron application.

Screenshot:

Folder Structure.

In the Project folder or the root of the app we will keep the package.json file, which have all the dependencies to run the application.Also the main.js file and any other application-wide files we need are kept int the root of the app..

The “app” folder will house our HTML files of various types within folders like css, js and img.

Judging by the file structure, you can never guess this is a desktop app and not just a simple website. This project follows the AirBNB Style guide to code.

Purely starting the main process doesn’t give the users of your application any UI windows.

Conclusion

Overall Electron is an exciting way to build desktop web applications using web technologies.

SUSI Desktop app is built on it. Single code can be used to make native apps for Windows / Linux and Mac OS X.

Continue ReadingSUSI Desktop App in Electron

Test SUSI Web App with Facebook Jest

Jest is used by Facebook to test all Javascript codes specially React code snippets. If you need to setup jest on your react application you can follow up these simple steps. But if your React application is made with “create-react-app”, you do not need to setup jest manually. Because it comes with Jest. You can run test using “react-scripts” node module.

Since SUSI chat is made with “create-react-app” we do not need to install and run Jest directly. We executed our test cases using “npm test” it executes “react-scripts test” command. It executes all “.js” files under “__tests__” folders. And all other files with “.spec.js” and “.test.js” suffixes.

React apps that are made from “create-react-app” come with sample test case (smoke test) and that checks whether whole application is built correctly or not. If it passes the smoke test then it starts to run further test cases.

import React from 'react';
import ReactDOM from 'react-dom';
import ChatApp from '../../components/ChatApp.react';
it('renders without crashing', () => {
 const div = document.createElement('div');
 ReactDOM.render( < ChatApp / > , div);
});

This checks all components which are inside of the “<ChatApp />” component and it check whether all these components integrate correctly or not.

If we need to check only one component in isolated environment. We can use shallow rendering API. we have used shallow rendering API to check each and every component in isolated environment.

We have to install enzyme and test renderer before use it.

npm install --save-dev enzyme react-test-renderer

import React from 'react';
import MessageSection from '../../components/MessageSection.react';
import { shallow } from 'enzyme';
it('render MessageListItem without crashing',()=>{
  shallow(<MessageSection />);
})

This test case tests only the “MessageListItem”, not its child components.

After executing “npm test” you will get the passed and failed number of test cases.

If you need to see the coverage you can see it without installing additional dependencies.

You just need to run this.

npm test -- --coverage

It will show the output like this.

This view shows how many lines, functions, statements, branches your program has and this shows how much of those covered from the test cases you have.

If we are going to write new test cases for susi chat, we have to make separate file in “__tests__” folder and name it with corresponding file name that we are going to test.

it('your test case description',()=>{
 //test what you need 
})

Normally test cases looks like this.in test cases you can use “test” instead of “it” .after test case description, there is a fat arrow function. In side this fat arrow function you can add what you need to test

In below example I have compared returned value of the function with static value.

function funcName(){
 return 1;
}

it('your test case description',()=>{
 expect(funcName()).toBe(1);
})

You have to add your function/variable that need to be tested into “expect()” and value you expect from that function or variable into “toBe()”.  Instead of “toBe()” you can use different functions according to your need.

If you have a long list of test cases you can group them ( using describe()).

describe('my beverage', () => {
  test('is delicious', () => {
    expect(myBeverage.delicious).toBeTruthy();
  });

  test('is not sour', () => {
    expect(myBeverage.sour).toBeFalsy();
  });
});

This post covers only few things of testing . You can learn more about jest testing from official documentation here.

Continue ReadingTest SUSI Web App with Facebook Jest

Adding a new Servlet/API to SUSI Server for Skill Wiki

Susi skill wiki is an editor to write and edit skill easily. It follows an API-centric approach where the Susi server acts as API server and a web front-end  act as the client for the API and provides the user interface. A skill is a set of intents. One text file represents one skill, it may contain several intents which all belong together.

The schema for storing a skill is as following:

Using this, one can access any skill based on four tuples parameters model, group, language, skill.  To achieve this on server side let’s create an API endpoint to list all skills based on given model, groups and languages. To check the source for this endpoint clone the susi_server repository from here.

git clone https://github.com/fossasia/susi_server.git

Have a look at documentation for more information about Susi Server.

The Servlet java file is placed in susi_server/ai/susi/server/api/cms/ListSkillService. To implement the endpoint we will use the HttpServlet class which provides methods, such as doGet and doPost, for handling HTTP-specific services.

In Susi Server an abstract class AbstractAPIHandler extending HttpServelets and implementing API handler interface is provided.  Next we will inherit our ListSkillService class from AbstractAPIHandler and implement APIhandler interface.

To implement our servlet we will be overriding 4 methods namely

  • Minimal Base User role 

public BaseUserRole getMinimalBaseUserRole() {
return BaseUserRole.ANONYMOUS;
}

This method tells the minimum Userrole required to access this servlet it can also be ADMIN, USER. In our case it is Anonymous. A User need not to log in to access this endpoint.

  • Default Permissions  

public JSONObject getDefaultPermissions(BaseUserRole baseUserRole) {
return null;
}

This method returns the default permission attached with base user role, our servlets has nothing to do with it, therefore we can simply return null for this case.

  • The API Path 

public String getAPIPath() {
return "/cms/getSkillList.json";
}

This methods sets the API endpoint path, it gets appended to base path which is 127.0.0.1:4000/cms/getSkillList.json for the local host and http://api.susi.ai/cms/getSkillList.json for the server.

  • The ServiceImpl method 

public ServiceResponse serviceImpl(Query call, HttpServletResponse response, Authorization rights, final JsonObjectWithDefault permissions) {

        String model_name = call.get("model", "general");
        File model = new File(DAO.model_watch_dir, model_name);
        String group_name = call.get("group", "knowledge");
        File group = new File(model, group_name);
        String language_name = call.get("language", "en");
        File language = new File(group, language_name);

        ArrayList fileList = new ArrayList();
        fileList =  listFilesForFolder(language, fileList);
        JSONArray jsArray = new JSONArray(fileList);

        JSONObject json = new JSONObject(true)
                .put("model", model_name)
                .put("group", group_name)
                .put("language", language_name)
                .put("skills", jsArray);
        return new ServiceResponse(json);

    }

    ArrayList listFilesForFolder(final File folder,  ArrayList fileList) {

        File[] filesInFolder = folder.listFiles();
        if (filesInFolder != null) {
            for (final File fileEntry : filesInFolder) {
                if (!fileEntry.isDirectory()) {
                    fileList.add(fileEntry.getName()+"");
                }
            }
        }
        return  fileList;
    }

To access any skill we need parameters model, group, language. We get this through call.get method where first parameter is the key for which we want to get the value and second parameter is the default value. Based on received model, group and language browse files in that folder and put them in Json array to return the Service Json response.
That’s all you have successfully implemented the API endpoint to list all the skills in given model group and language. Move on and test it.

And see the resultsTo access any skill we need  parameters model, group, language. We get this through call.get method where first parameter is the key for which we want to get the value and second parameter is the default value. Based on received model, group and language browse files in that folder and put them in Json array to return the Service Json response.

 You have  successfully implemented the API endpoint to list all the skills in given model group and language. Move on and test it.

It’s easy isn’t it you have learnt how to add an endpoint in the server, it’s time to go ahead and create more endpoints to enhance Susi Server, take a look and contribute to Susi Server.

Continue ReadingAdding a new Servlet/API to SUSI Server for Skill Wiki

How to make SUSI AI Line Bot

In order to integrate SUSI’s API with Line bot you will need to have a line account first so that you can follow below procedure. You can download app from here.

Pre-requisites:

  • Line app
  • Github
  • Heroku

    Steps:
    1. Install Node.js from the link below on your computer if you haven’t installed it already https://nodejs.org/en/.
    2. Create a folder with any name and open shell and change your current directory to the new folder you created.
    3. Type npm init in command line and enter details like name, version and entry point.
    4. Create a file with the same name that you wrote in entry point in above given step. i.e index.js and it should be in same folder you created.
    5. Type following commands in command line  npm install –save @line/bot-sdk. After bot-sdk is installed type npm install –save express after express is installed type npm install –save request when all the modules are installed check your package.json modules will be included within dependencies portion.

      Your package.json file should look like this.

      {
      "name": "SUSI-Bot",
      "version": "1.0.0",
      "description": "SUSI AI LINE bot",
      "main": "index.js",
      "dependencies": {
         "@line/bot-sdk": "^1.0.0",
         "express": "^4.15.2",
         "request": "^2.81.0"
      },
      "scripts": {
         "start": "node index.js"
       }
      }
    6. Copy following code into file you created i.e index.js
      'use strict';
      const line = require('@line/bot-sdk');
      const express = require('express');
      var request = require("request");
      
      // create LINE SDK config from env variables
      
      const config = {
         channelAccessToken: process.env.CHANNEL_ACCESS_TOKEN,
         channelSecret: process.env.CHANNEL_SECRET,
      };
      
      // create LINE SDK client
      
      const client = new line.Client(config);
      
      
      // create Express app
      // about Express: https://expressjs.com/
      
      const app = express();
      
      // register a webhook handler with middleware
      
      app.post('/webhook', line.middleware(config), (req, res) => {
         Promise
             .all(req.body.events.map(handleEvent))
             .then((result) => res.json(result));
      });
      
      // event handler
      
      function handleEvent(event) {
         if (event.type !== 'message' || event.message.type !== 'text') {
             // ignore non-text-message event
             return Promise.resolve(null);
         }
      
         var options1 = {
             method: 'GET',
             url: 'http://api.asksusi.com/susi/chat.json',
             qs: {
                 timezoneOffset: '-330',
                 q: event.message.text
             }
         };
      
         request(options, function(error, response, body) {
             if (error) throw new Error(error);
             // answer fetched from susi
             //console.log(body);
             var ans = (JSON.parse(body)).answers[0].actions[0].expression;
             // create a echoing text message
             const answer = {
                 type: 'text',
                 text: ans
             };
      
             // use reply API
      
             return client.replyMessage(event.replyToken, answer);
         })
      }
      
      // listen on port
      
      const port = process.env.PORT || 3000;
      app.listen(port, () => {
         console.log(`listening on ${port}`);
      });
    7. Now we have to get channel access token and channel secret to get that follow below steps.

    8. If you have Line account then move to next step else sign up for an account and make one.
    9. Create Line account on  Line Business Center with messaging API and follow these steps:
    10. In the Line Business Center, select Messaging API under the Service category at the top of the page.
    11. Select start using messaging API, enter required information and confirm it.
    12. Click [email protected] Manager option, In settings go to bot settings and Enable messaging API
    13. Now we have to configure settings. Allow messages using webhook and select allow for “Use Webhooks”.
    14. Go to Accounts option at top of page and open LINE Developers.
    15. To get Channel access token for accessing API, click ISSUE for the “Channel access token” item.
    16. Click EDIT and set a webhook URL for your Channel. To get webhook url deploy your bot to heroku and see below steps.
    17. Before deploying we have to make a github repository for chatbot to make github repository follow these steps:

      In command line change current directory to folder we created above and  write

      git init
      git add .
      git commit -m”initial”
      git remote add origin <URL for remote repository> 
      git remote -v
      git push -u origin master 

      You will get URL for remote repository by making repository on your github and copying this link of your repository.

    18. To deploy your bot to heroku you need an account on Heroku and after making an account make an app.
    19. Deploy app using github deployment method.


    20. Select Automatic deployment method.


    21. After making app copy this link and paste it in webhook url in Line channel console page from where we got channel access token.

                https://<your_heroku_app_name>.herokuapp.com/webhook
    22. Your SUSI AI Line bot is ready add this account as a friend and start chatting with SUSI.
      Here is the LINE API reference https://devdocs.line.me/en/
Continue ReadingHow to make SUSI AI Line Bot

Skills for SUSI

Susi is an open source personal assistant can do a lot of amazing things for you apart from just answering queries in the text. Susi supports many action type such as answer, table, pie chart, RSS, web search, map. Actions contain a list of objects each with type attribute, there can be more than one actions in the object list. For example

curl http://api.susi.ai/susi/chat.json?timezoneOffset=-330&q=Who+are+you

We get a json response similar to

{
"query": "Who are you",
"answers": [{
"data": [],
"metadata": {"count": 0},
"actions": [{
"type": "answer",
"expression": "I was told that I am a Social Universal Super Intelligence. What do you think?"
}]
}],
}

The above query is an example of action type ‘answer’, for developing more skills on action type answer refer to the Fossasia  blog : How to teach Susi skills.  In this blog we will see how to teach a table skill to susi and how susi interprets the skill .So let’s add a skill to display football teams and its players using service Football-Data.org .

For writing rules Open a new etherpad with a desired name <etherpad name> at http://dream.susi.ai/

  1. Next let’s set some query for which we want Susi to answer.

Example queries:

tell me the teams in premier league | Premier league teams

To get answer we define the following rule

!console:
{
"url":"http://api.football-data.org/v1/competitions/398/teams",
"path":"$.teams",
"actions":[{
"type":"table",
"columns":{"name":"Name","code":"Code","shortName":"Short Name","crestUrl":Logo},
"count": -1
}]
}
eol

Expalanation:

The JSON response for above URL look like this:

{
"_links": {
"self": {
"href": "http://api.football-data.org/v1/competitions/398/teams"
},
"competition": {
"href": "http://api.football-data.org/v1/competitions/398"
}
},
"count": 20,
"teams": [{
"_links": {
"self": {
"href": "http://api.football-data.org/v1/teams/66"
},
"fixtures": {
"href": "http://api.football-data.org/v1/teams/66/fixtures"
},
"players": {
"href": "http://api.football-data.org/v1/teams/66/players"
}
},
"name": "Manchester United FC",
"code": "MUFC",
"shortName": "ManU",
"squadMarketValue": null,
"crestUrl": "http://upload.wikimedia.org/wikipedia/de/d/da/Manchester_United_FC.svg"
},
{
"_links": {
"self": {
"href": "http://api.football-data.org/v1/teams/65"
},
"fixtures": {
"href": "http://api.football-data.org/v1/teams/65/fixtures"
},
"players": {
"href": "http://api.football-data.org/v1/teams/65/players"
}
},
"name": "Manchester City FC",
"code": "MCFC",
"shortName": "ManCity",
"squadMarketValue": null,
"crestUrl": "https://upload.wikimedia.org/wikipedia/en/e/eb/Manchester_City_FC_badge.svg"
}]
}

The attribute ‘path’  statuses object contains a list of objects for which we want to show the table. The table is defined with the action type “table” and a columns object which provides a mapping from the column value names to the descriptive names that will be rendered in the client’s output. In our case, there are  4 columns  Name of the team, Team Code, Short Name of the team, and the Team logo’s URL.

The count attribute is used to denote how many rows to populate the table. If count = -1 , then it means as many as possible or displays all the results.It’s easy, isn’t it? We have successfully created table skill for SUSI. Let’s try it out

Go to http://chat.susi.ai/ and type: dream < your dream name>. And type your queries like  “tell me the teams in premier league” in our case  and see how SUSI presents you the results


Next, want to create some more skills. Let’s teach SUSI to play a game Rock, Paper, Scissors, Lizard, Spock. SUSI can give random answers to your queries for example

What is your favorite dish
Potatoes|Vegetables|Fish

SUSI will return any random answer “Potatoes, “Vegetables” or “Fish”. We would use this to create a game.
First form a general query like “I am getting bored let us play something” and the add an answer to it by adding the rules of the game something like

I am getting bored let us play something
Let's play the game Rock, Paper, Scissors, Lizard, Spock. It is an expansion to the game Rock, Paper, Scissors. We both will  pick a variable and reveal. The winner will be  one who defeats the others. It goes like Scissors cuts Paper,Paper covers Rock, Rock crushes Lizard, Lizard poisons Spock, Spock smashes Scissors, Scissors decapitates Lizard, Lizard eats Paper, Paper disproves Spock, Spock vaporizes Rock, (and as it always has) Rock crushes Scissors. Let's begin, choose something

Next since user would choose between any 5 of the Rock, Scissors, Lizard,Paper and spock add answers to each of them based on the rules.

Rock | I choose Rock
Paper, Paper covers Rock I won :) | Sccisors, Rock crushes Scissors , I lost :( | Lizard, Rock crushes Lizard , You won :-) | Spock, Spock vaporises Rock, You lost :) | Rock, Ahh It's a Tie :-D

Paper | I choose Paper
Rock, Paper covers Rock You won :( | Sccisors, Scissors cuts Paper I won :) | Lizard, Lizard eats Paper I won :) | Spock , Paper disproves Spock You won :( | Paper, Ahh It's a Tie :-D

Lizard | I choose Lizard
Rock, Rock crushes Lizard , I won :-) | Spock, Lizard poisons Spock You won :( | Sccisors, Scissors decapitates Lizard I won :) | Paper, Lizard eats Paper You won :( | Lizard, Ahh It's a Tie :-D

Scissors | I choose Scissors
Rock, Rock crushes Scissors , You lost :) | Paper , Scissors cuts Paper You won :( | Spock, Spock smashes Scissors I won :) | Lizard, Scissors decapitates Lizard You won :( | Scissors, Ahh It's a Tie :-D

Spock | I choose Spock
Rock, Spock vaporizes Rock, I lost :( | Lizard, Lizard poisons Spock You lost :) | Scissors, Spock smashes Scissors I lost :(| Paper, Paper disproves Spock I won :) | Spock, Ahh It's a Tie :-D

SUSI will return any random answer to the variable picked go and try to beat SUSI.

Have fun with SUSI and do create some other awesome skills for SUSI, read this tutorial  for more details.

Continue ReadingSkills for SUSI

Hotword Detection with Pocketsphinx for SUSI.AI

Susi has many apps across all the major platforms. Latest addition to them is the Susi Hardware which allows you to setup Susi on a Hardware Device like Raspberry Pi.

Susi Hardware was able to interact with a push of a button, but it is always cool and convenient to call your assistant anytime rather than using a button.

Hotword Detection helps achieve that. Hotword Detection involves running a process in the background that continuously listens for voice. On noticing an utterance, we need to check whether it contains desired word. Hotword Detection and its integration with Susi AI can be explained using the diagram below:

 

What is PocketSphinx?

PocketSphinx is a lightweight speech recognition engine, specifically tuned for handheld and mobile devices, though it works equally well on the desktop.

PocketSphinx is free and open source software. PocketSphinx has various applications but we utilize its power to detect a keyword (say Hotword) in a verbally spoken phrase.

Official Github Repository: https://github.com/cmusphinx/pocketsphinx

Installing PocketSphinx

We shall be using PocketSphinx with Python. Latest version on it can be installed by

pip install pocketsphinx

If you are using a Raspberry Pi or ARM based other board, Python 3.6 , it will install from sources by the above step since author doesn’t provide a Python Wheel. For that, you may need to install swig additionally.

sudo apt install swig

How to detect Hotword with PocketSphinx?

PocketSphinx can be used in various languages. For Susi Hardware, I am using
Python 3.

Steps:

      • Import PyAudio and PocketSphinx
        from pocketsphinx import *
        import pyaudio
        

         

      • Create a decoder with certain model, we are using en-us model and english us default dictionary. Specify a keyphrase for your application, for Susi AI , we are using “Susi” as Hotword
        pocketsphinx_dir = os.path.dirname(pocketsphinx.__file__)
        model_dir = os.path.join(pocketsphinx_dir, 'model')
        
        config = pocketsphinx.Decoder.default_config()
        config.set_string('-hmm', os.path.join(model_dir, 'en-us'))
        config.set_string('-keyphrase', 'susi')
        config.set_string('-dict', os.path.join(model_dir, dict_name))
        config.set_float('-kws_threshold', self.threshold)

         

      • Start a PyAudio Stream from Microphone Input
        p = pyaudio.PyAudio()
        
        stream = p.open(format=pyaudio.paInt16, channels=1, rate=16000, input=True, frames_per_buffer=20480)
        stream.start_stream()

         

      • In a forever running loop, read from stream and process buffer in chunks of 1024 frames if it is not empty.
        buf = stream.read(1024)
        if buf:
            decoder.process_raw(buff)
        else:
            break;
        

         

      • Check for hotword and start speech recognition if hotword detected. After returning from the method, start detection again.

        if decoder.hyp() is not None:
            print("Hotword Detected")
            decoder.end_utt()
            start_speech_recognition()
            decoder.start_utt()
        

         

      • Run the app, if detection doesn’t seem to work well, adjust kws_threshold in step 2 to give optimal results.

In this way, Hotword Detection can be added to your Python Project. You may also develop some cool hacks with our AI powered Assistant Susi by Voice Control.
Check repository for more info: https://github.com/fossasia/susi_hardware

Continue ReadingHotword Detection with Pocketsphinx for SUSI.AI

Displaying Web Search and Map Preview using Glide in SUSI Android App

SUSI.AI has many skills. Two of which are displaying web search of a certain query and displaying a map of certain position. This post will cover how these things are implemented in Susi Android App and how a preview is displayed using Bumptech’s free and open source Glide library. So, what is glide? According to Glide docs, Glide is a fast and efficient open source media management and image loading framework for Android that wraps media       decoding, memory and disk caching, and resource pooling into a simple and easy to use interface.

Now, lets describe how this framework is used in Susi App. So, let’s cover it’s two uses one by one :

Map Preview :

    

Whenever a user queries something like “Where is Singapore” or “Where am I”, the response from server is a map with latitude, longitude and zoom level. See the “type” which is map. It indicates android app that it needs to display a map with provided values.

“actions”: [
     {
       “type”: “answer”,
       “expression”: “Singapore is a place with a population of 3547809. Here is a map: https://www.openstreetmap.org/#map=13/1.2896698812440377/103.85006683126556”
     },
     {
       “type”: “anchor”,
       “link”: “https://www.openstreetmap.org/#map=13/1.2896698812440377/103.85006683126556”,
       “text”: “Link to Openstreetmap: Singapore”
     },
     {
       “type”: “map”,
       “latitude”: “1.2896698812440377”,
       “longitude”: “103.85006683126556”,
       “zoom”: “13”
     }
   ]
 }]

So, these values are then plugged into the below mentioned url where length x width is size of image to be retrieved. This url links to an image of the map with center and size as defined.

http://staticmap.openstreetmap.de/staticmap.php?center=latitude,longitude&zoom=zoom&size=lengthxwidth

So, now as we have a link to the image to be displayed. We just need something to get the image from that link and display it in the app. That’s where Glide comes to rescue. It loads the image from the link to an ImageView.

Glide.with(currContext).load(mapHelper.getMapURL()).listener(new RequestListener<String, GlideDrawable>() {

@Override
public boolean onException(Exception e, String model, Target<GlideDrawable> target, boolean isFirstResource) {
return false;
}

@Override
public boolean onResourceReady(GlideDrawable resource, String model,
Target<GlideDrawable> target, boolean isFromMemoryCache, boolean isFirstResource){
mapViewHolder.pointer.setVisibility(View.VISIBLE);
return false;
}

}).into(mapViewHolder.mapImage);

So, what the above code does is that it displays image from url mapHelper.getMapURL() to mapViewHolder.mapImage.

So, by this way, we are able to display a preview of the map in the chat itself. The user can then click on the image to load the         complete map.

Web Search Preview :

When a user enters queries like “search for dog”, the response from server is a websearch with the query to be searched,         something like this.

“actions”: [
     {
       “type”: “answer”,
       “expression”: “Here is a web search result:”
     },
     {
       “type”: “websearch”,
       “query”: “dog”
     }
   ]

So, now we know the query to be searched, we can use any search api to get results about that query ad display it in the app. In Susi android app, we have used DuckDuckGo open source api to do that task. We call this url

https://api.duckduckgo.com/?format=json&pretty=1&q=query&ia=meanings

which gives a json response with results. We then use the json     results which contains the link to result, image to be displayed and a short abstract of the result.

{
“Result” : “<a href=\”https://duckduckgo.com/Dog\”>Dog</a>A member of genus Canis that forms part of the wolf-like canids, and is the most widely abundant…”,
“Icon” : {
“URL” : “https://duckduckgo.com/i/16101b42.jpg”,
“Height” : “”,
“Width” : “”
},
“FirstURL” : “https://duckduckgo.com/Dog”,
“Text” : “Dog A member of genus Canis that forms part of the wolf-like canids, and is the most widely abundant…”
}

There are three things we are displaying in the websearch preview, which are Title, text and an image. We display the query as the title, text in json response as text and use glide to get the image from the url provided in the the json response.

glide.load(url)
.crossFade()
.into(websearchholder.previewImageView);

Using the above code, we load the image from the image url to websearchholder.previewImageView().

So, by this way, we are able to display a preview of the search     result. The user can then touch on this preview which opens the     detailed page of the result.

Happy Coding!

Continue ReadingDisplaying Web Search and Map Preview using Glide in SUSI Android App

Conversion of CSS styles into React styles in SUSI Web Chat App

Earlier this week we had an issue where the text in our search box of the SUSI web app was not white even after having all the required styles. After careful inspection it was found that there is a attribute named -webkit-text-fill-color which was set to black.

Now I faced this issue as adding such attribute to our reactJs code will cause lint errors. So after careful searching stackoverflow, i found a way to add css attribute to our react code by using different case. I decided to write a blog on this for future reference and it might come handy to other developers as well.

If you want to write css in javascript, you have to turn dashed-key-words into camelCaseKeys

For example:

background-color => backgroundColor
border-radius => borderRadius
but vendor prefix starts with capital letter (except ms)
-webkit-box-shadow => WebkitBoxShadow (capital W)
-ms-transition => msTransition ('ms' is the only lowercase vendor prefix)

const containerStyle = {
  WebkitBoxShadow: '0 0 0 1000px white inset'
};

So in our case:-

-webkit-text-fill-color became WebkitTextFillColor

The final code of styles looked like: –

const searchstyle = {
      WebkitTextFillColor: 'white',
      color: 'white'
    }

Now, because inline styles gets attached on tags directly instead of using selectors, we have to put this style on the <input> tag itself, not the container.

See the react doc #inline-styles section for more details.

Continue ReadingConversion of CSS styles into React styles in SUSI Web Chat App