Giving a Voice to Susi

Susi AI already has various apps and is available as a chatbot in various messaging platforms. We are going a step forward to make an SDK available for Susi that can be integrated on any Hardware Device (say speakers, toys, your bicycle etc – possibilities are endless )

One of the problem that I encountered while making a Prototype for the same is selecting an appropriate Text to Speech (TTS)  Engine.

It was a challenge, since on platforms like Android and iOS , you may utilize TTS engines bundled with Platform easily via a platform specific API, which are well optimized and give good performance.  The same was difficult on a hardware device that can run only Linux with no TTS provided by default.

Thus, I explored some possibilities

eSpeak TTS: eSpeak TTS (http://espeak.sourceforge.net/) was the first option considered for the task.
eSpeak is a compact open source software speech synthesizer for English and other languages, for Linux and Windows.

The major advantage of eSpeak is its small size (2MB) and small memory footprint which is advantageous in Low Memory Hardware like Orange Pi Zero or Raspberry Pi Zero.


Setting up eSpeak was easy but with its advantages , there were some drawbacks too.

  • The voice synthesis was quite robotic.
  • Very few voices were available.

Festival TTS: Festival offers a general framework for building speech synthesis systems as well as including examples of various modules. As a whole it offers full text to speech through a number APIs: from shell level, though a Scheme command interpreter, as a C++ library, from Java, and an Emacs interface.


Festival is free software. Festival and the speech tools are distributed under an X11-type licence allowing unrestricted commercial and non-commercial use alike.

Installing Festival:

On Arch Linux , it was pretty straight forward.

sudo pacman -S festival

There is a full wiki dedicated to it. ( https://wiki.archlinux.org/index.php/Festival )

Testing Festival

Festival has an interpreter to test it out. It can be invoked using
You may test out a TTS output using:

festival> (SayText "Hi!! I am Susi")

But the default sound in festival is still robotic and male. You don’t want your Personal Assistant to scare you out when you speak to her.

Thus, I searched on what are the best female voices available for Festival.

After looking at a discussion on the thread, https://ubuntuforums.org/showthread.php?t=751169 ,

I found that CMU-Arctic and HTS are some of the best voice sets for  Festival.

In Arch Linux, additional voice packs, are supplied in two additional packages,

festival-us and festival-english

Installation is straightforward:

sudo pacman -S festival-us festival-english

Now, on festival REPL , we can test out our new voices.

To see all available voices

festival> (voice.list)
(rab_diphone kal_diphone cmu_us_rms_cg cmu_us_awb_cg cmu_us_slt_cg)

Testing out a voice

festival> (voice_cmu_us_awb_cg)
cmu_us_awb_cg
festival> (SayText "Hi!! I am Susi")

This way after testing out all voices, with many different phrases. cmu_us_slt_cg  felt like an appropriate voice.

Setting Voice as Default

Voice may be set as default by adding following line to .festivalrc

 (set! voice_default voice_cmu_us_slt_cg)

Now festival will be invoked with this voice as default.

You may read a file using festival using

$ festival --tts <filename>

Calling Festival from Python

This was accomplished by first writing to a file, and then calling a subprocess to output speech using festival tts.

def speak(text):
    filename = '.response'
    file=open(filename,'w')
    file.write(text)
    file.close()
    # Call festival tts to reply the response by Susi
    subprocess.call('festival --tts '+filename, shell=True)

 

So, festival works pretty well on a moderately powerful machine, but while trying it on Raspberry Pi with a custom voice, there was noticeable amount of time delay for synthesis, so I searched for more alternatives.

Flite TTS: CMU Flite (festival-lite) is a small, fast run-time open source text to speech synthesis engine developed at CMU and primarily designed for small embedded machines and/or large servers. Flite is designed as an alternative text to speech synthesis engine to Festival for voices built using the FestVox suite of voice building tools.
Flite gives a noticeable speech improvement. It gives a Festival like output for 1/10th the amount taken by Festival.

Installing Flite TTS

On Raspberry Pi (Raspbian) , the version in Raspbian Jessie Repository is Flite 1.4 which does not support using an external speech file, so we need to compile and install from sources.

$ wget http://www.festvox.org/flite/packed/latest/flite-2.0.0-release.tar.bz2
$ tar xf flite-2.0.0-release.tar.bz2
$ cd flite-2.0.0-release/
$ ./configure
$ make
$ sudo make install

Now, you may download flitevox file for voice you wish to use from

http://www.festvox.org/flite/packed/latest/voices/

It can be invoked using

$ flite -voice file://<flitevox_file_path> -f <filepath-to-read>

You may save output audio stream to a file using

$ flite -voice file://<flitevox_file_path> -f <filepath-to-read> -o output.wav

Now, you can playback the audio. This can be invoked from python using.

import os

def speak_flite_tts(text):
    filename = '.response'
    file = open(filename, 'w')
    file.write(text)
    file.close()
    # Call flite tts to reply the response by Susi
    flite_speech_file = 'cmu_us_slt.flitevox'
    print('flite -voice file://{0} -f {1}'.format(flite_speech_file, filename))
    os.system('flite -v -voice file://{0} -f {1} -o output.wav'.format(flite_speech_file, filename))
    os.system('aplay output.wav')

 

In this way, we added offline Text to Speech Support of Susi Hardware SDK.

Continue ReadingGiving a Voice to Susi

Implementing a chatbot using the SUSI.AI API

SUSI AI is an intelligent Open Source personal assistant. It is a server application which is able to interact with humans as a personal assistant. The first step in implementing a bot using SUSI AI is to specify the pathway for query response from SUSI AI server.

The steps mentioned below provide a step-by-step guide to establish communication with SUSI AI server:

    1. Given below is HTML code that demonstrates how to connect with SUSI API through an AJAX call. To put this file on a Node Js server, see Step 2.  To view the response of this call, follow Step 4.

      <!DOCTYPE html>
      <body>
      <h1>My Header</h1>
      <p>My paragraph.</p>
      //Script with source here 
      //Script to be written here 
      </body>
      </html>
      

      In above code add scripts given below and end each script with closing tag </script>. In the second script we are calling SUSI API with hello query and showing data that we are receiving through call on console.

      <script src="https://ajax.googleapis.com/ajax/libs/jquery/3.2.1/jquery.min.js">
    2. <script>
      $(function (){ 
          $.ajax({ 
              dataType: 'jsonp', 
              type:'GET', url: 'http://api.susi.ai/susi/chat.json? timezoneOffset=-300&q=hello', 
               success: function(data){ 
                   console.log('success', data); 
               } 
          }); 
      });
    3. Code below is in node js to setup localhost and getting the same above result on browser. Below is Node Js code to setup a server at localhost for the above created HTML file.

      var http = require('http');
       var fs = require('fs');
       http.createServer(function (req, res) {
        fs.readFile('YOURFILENAME.html', function(err, data) {
          res.writeHead(200, {'Content-Type': 'text/html'});
          res.write(data);
          res.end();
        });
       }).listen(9000);
    4. We will get following response by running this Node js code and checking results on http://localhost:9000/ To run this code install Node Js and write “node filename.js” in command line.
    5. You can open above window by right clicking on page and selecting Inspect. Go to the Network option and select the relevant api call from left section of Inspect window.

We have successfully got response from SUSI API and now we can use this response for building bots for receiving replies for user.

Continue ReadingImplementing a chatbot using the SUSI.AI API

Hotword Detection in Susi Android

Hotword detection is one of the coolest features in Android. Voice control has emerged as a popular method for interacting with smartphones and wearable devices. It allows the user to interact with the app without even touching the device in a much intuitive way. At the same time it’s difficult to implement them in the Android app and requires use of large number of resources. Let us dive deeper into this topic.

So, Firstly What is a Hotword?

Hotword is generally a phrase or a word that can be used to initiate a particular task in the app. The user needs to speak the phrase or the hotword which on detection makes a callback that can be used to carry out a particular task in the app.

Examples of hotword include “Ok Google” which is used to initiate Google assistant in the Android mobile phones.

Why is it difficult to implement hotword detection?

To enable hotword detection is a cumbersome task. There are different problems associated with it including:-

  1. Change in the accent of the speaker.
  2. Continuous task of recognizing the voice and then matching it with the hotword.
  3. The features needs to be run as a service in android which causes the drain of battery.
  4. Also it is a memory intensive task at the same time.

Let’s us look at the present techniques available for the hotword detection.

  1. The Android provides a class called as AlwaysOnHotwordDetector. This class can be used to detect the keywords inside the activity of an Android app. For more details you can visit this link.

The implementation goes like this

/**

* @param text The keyphrase text to get the detector for.

* @param locale The java locale for the detector.

* @param callback A non-null Callback for receiving the recognition events.

* @param voiceInteractionService The current voice interaction service.

* @param modelManagementService A service that allows management of sound models.

*

* @hide

*/


public AlwaysOnHotwordDetector(String text, Locale locale, Callback callback, KeyphraseEnrollmentInfo keyphraseEnrollmentInfo, IVoiceInteractionService voiceInteractionService, IVoiceInteractionManagerService modelManagementService) {

  mText = text;

  mLocale = locale;

  mKeyphraseEnrollmentInfo = keyphraseEnrollmentInfo;

  mKeyphraseMetadata = mKeyphraseEnrollmentInfo.getKeyphraseMetadata(text, locale);

  mExternalCallback = callback;

  mHandler = new MyHandler();

  mInternalCallback = new SoundTriggerListener(mHandler);

  mVoiceInteractionService = voiceInteractionService;

  mModelManagementService = modelManagementService;

  new RefreshAvailabiltyTask().execute();



}

2.  We can also detect hotword using Pocketsphinx library in Android. The library acts as a service in the Android app. It is quite efficient when it comes to battery consumption. The implementation goes like this:-    

recognizer = defaultSetup()
   .setAcousticModel(new File(assetsDir, "en-us-ptm"))
   .setDictionary(new File(assetsDir, 
"cmudict-en-us.dict"))
   .getRecognizer();
recognizer.addListener(this);



// Create keyword-activation search.
recognizer.addKeyphraseSearch(KWS_SEARCH, KEYPHRASE);

// Create grammar-based searches.
File menuGrammar = new File(assetsDir, "menu.gram");
recognizer.addGrammarSearch(MENU_SEARCH, menuGrammar);

// Next search for digits
File digitsGrammar = new File(assetsDir, "digits.gram");
recognizer.addGrammarSearch(DIGITS_SEARCH, digitsGrammar);

// Create language model search.
File languageModel = new File(assetsDir, "weather.dmp");
recognizer.addNgramSearch(FORECAST_SEARCH, languageModel);


recognizer.startListening(searchName);

Now Lets us look at how we are implementing it in Susi Android:

In Susi Android, we have used CMU’s Pocketsphinx library for the hotword detection. We are using “Hi Susi” as the hotword for the detection.

private SpeechRecognizer recognizer;

/* Keyword we are looking for to activate menu */

private static final String KEYPHRASE = "hi susi";

/* Named searches allow to quickly reconfigure the decoder */

private static final String KWS_SEARCH = "hi susi";



The function setupRecognizer is used to initialize the Recognizer and detect the keyphrase



private void setupRecognizer(File assetsDir) throws IOException {

// The recognizer can be configured to perform multiple searches

// of different kind and switch between them

recognizer = SpeechRecognizerSetup.defaultSetup()

.setAcousticModel(new File(assetsDir, "en-us-ptm"))

.setDictionary(new File(assetsDir, "cmudict-en-us.dict"))

.setRawLogDir(assetsDir)

.getRecognizer();

recognizer.addListener(this);

recognizer.addKeyphraseSearch(KWS_SEARCH, KEYPHRASE);

There is still some need of improvement towards this approach in Susi app as the success rate of detection of keyword is not very high. We can overcome this by training the model or having a set of strings similar to that of “Hi Susi” such as “Hii Susi” or “Hi Sushi”. This will increase the rate of detection.  

Continue ReadingHotword Detection in Susi Android

Deploying SUSI.AI with Docker

Docker is much more efficient than VM in allocating shared resources between various containers as shown in figure. To deploy SUSI we need to create docker container. There are two ways to build it. First way is fork the SUSI project in github. Then you can signup in dockerhub and create autobuild docker container. The second way is to manually build docker file from command prompt of your computer. The following instructions needs to be executed in cloud shell or linux machine.

sudo apt-get update
sudo apt-get upgrade
sudo apt-get -y install docker.io
sudo fallocate -l 4G /swapfile
sudo chmod 600 /swapfile
sudo mkswap /swapfile
sudo swapon /swapfile
sudo docker build https://github.com/fossasia/susi_server.git


The first three commands install docker software to the machine. Next three lines give required permissions and execution abilities to docker. Final command builds docker container from github. Thus, We have successfully made docker container. We can deploy it on the cloud by using following command.

sudo docker run -d -p 80:80 -p 443:443 susi


Deploying Susi on cloud with kubernetes

We will use Google Cloud Service platform for demonstration. Create your GCS account. Goto dashboard and click on compute engine.

Enable Billing and Create New Project named XYZ. Open the terminal, by clicking the Google cloud shell button on the top.

Please set the compute zone to your nearest zone by running the below command.

gcloud config set compute/zone us-central1-a


Now we need to create cluster, on which we deploy susi app. We do it by this command.

gcloud container clusters create hello-cluster --num-nodes=3


We need to get docker from dockerhub and push it to our project repo.We do it by these commands.

sudo docker pull jyothiraditya/susi_server
gcloud docker -- push <image-id> gcr.io/<project-id>/<name>

We run the docker image on cluster by following commands.

kubectl run susi-server --image=gcr.io/<project-id>/<name> --port=80
kubectl get pods

We expose the container to external traffic, with help of load balancer by the following command.

kubectl expose deployment susi-server --type="LoadBalancer --port=80"


We get the external ip-address to access susi from browser. By entering

kubectl get service susi-server


You can now view the app by going to “EXTERNAL-IP:80”.

References : Docker build , Kubernetes deployment, Google cloud deployment 

 

Continue ReadingDeploying SUSI.AI with Docker

Control flow of SUSI AI on Android and database management using Realm

While developing a chat based android application, one of the most important things is keeping track of user’s messages. Since the user might want to access them in the absence of Internet connectivity (i.e remotely) as well, storing them locally is also important.

In SUSI we are using Realm to keep things organized in a systematic manner and constructing model (or adding appropriate attributes) for every new data type which the application needs. Right now we have three main models namely ChatMessage, WebLink and WebSearchModel. These three java classes define the structure of each possible message.  ChatMessage evaluates and classifies incoming response from server either to be an image or map or pie chart or web search url or other valid types of response. WebSearchModel and WebLink models are there to manage those results which contains link to various web searches.

Various result based lists are maintained for smooth flow of application. Messages sent in absence of Internet are stored in a list – nonDelivered. All the messages have an attribute isDelivered which is set to true if and only if they have been queried, otherwise the attribute is set to false which puts it in the nonDelivered list. Once the phone is connected back to the internet and the app is active in foreground, the messages are sent to server, queried and we get the response back in the app’s database where the attributes are assigned accordingly.

 

I will explain a functionality below that will give a more clear view about our coding practices and work flow.

When a user long taps a message, few options are listed(these actions are defined in recycleradapters->ChatFeedRecyclerAdapter.java) from which you may select one. In the code, this triggers the method onActionsItemClicked(). In this Overridden method, we handle what happens when a user clicks on one of the following options from item menu. In this post I’ll be covering only about the star/important message option.

case R.id.menu_item_important:
    nSelected = getSelectedItems().size();
    if (nSelected >0)
    {
        for (int i = nSelected - 1; i >= 0; i--) 
        {
            markImportant(getSelectedItems().get(i));
        }
        if(nSelected == 1) 
        {
            Toast.makeText(context,nSelected+" message 
            marked 
            important",Toast.LENGTH_SHORT).show();
        } 
        else 
        {
            Toast.makeText(context, nSelected + " 
            messages marked important",                                                      
            Toast.LENGTH_SHORT).show();
        }
        Important = realm.where(ChatMessage.class).
        equalTo("isImportant",true)
        .findAll().sort("id");
        for(int i=0;i<important.size();++i)
            Log.i("message ","" + 
            important.get(i).getContent());
        Log.i("total ",""+important.size());
        actionMode.finish();
    }
return true;

We have the count of messages which were selected. Each message having a unique id is looped through and the attribute “isImportant” of each message object is modified accordingly. To modify this field, We call the method markImportant() and pass the id of message which has to be updated.

public void markImportant(final int position) {
    realm.executeTransaction(new Realm.Transaction() {
        @Override
        public void execute(Realm realm) {
            ChatMessage chatMessage = getItem(position);
            chatMessage.setIsImportant(true);
            realm.copyToRealmOrUpdate(chatMessage);
        }
    });
}

This method copies the instance of the message whose id it has received and updates the attribute “isImportant” and finally updates the message instance in the database.

Below given is the code for ImportantMessage activity which will help you understand properly how lists are used to query the database.

public class ImportantMessages extends AppCompatActivity {
 
    private Realm realm;
    private RecyclerView rvChatImportant;
    @Override
    protected void onCreate(Bundle savedInstanceState) {
       super.onCreate(savedInstanceState);
       realm = Realm.getDefaultInstance();
       rvChatImportant = (RecyclerView) findViewById 
                          (R.id.rv_chat_important);
       actionBar.setDisplayHomeAsUpEnabled(true);
       setChatBackground();
       setupAdapter();
 
        //call to other methods
    }
 
    private void setupAdapter() {
        rvChatImportant = (RecyclerView) findViewById 
                          (R.id.rv_chat_important);
        LinearLayoutManager linearLayoutManager = new 
                            LinearLayoutManager(this);
        linearLayoutManager.setStackFromEnd(true);
        rvChatImportant. 
        setLayoutManager(linearLayoutManager);
        rvChatImportant.setHasFixedSize(false);
        RealmResults<ChatMessage> importantMessages = 
        realm.where(ChatMessage.class). 
        equalTo("isImportant",true).findAll().sort("id");
        TextView tv_msg = (TextView) findViewById 
                          (R.id.tv_empty_list);
 
        if(importantMessages.size()!=0)
            tv_msg.setVisibility(View.INVISIBLE);
        else
            tv_msg.setVisibility(View.VISIBLE);
 
        ChatFeedRecyclerAdapter recyclerAdapter = new 
             ChatFeedRecyclerAdapter(Glide.with(this), this, 
             importantMessages, true);
        rvChatImportant.setAdapter(recyclerAdapter);
        rvChatImportant.addOnLayoutChangeListener(new 
        View.OnLayoutChangeListener() {
            @Override
            public void onLayoutChange(View view, int left, 
            int top, int right, int bottom,
            int oldLeft, int oldTop, int oldRight, int 
            oldBottom) {
                if (bottom < oldBottom) {
                    rvChatImportant.postDelayed(new 
                    Runnable() {
                        @Override
                        public void run() {
                            int scrollTo = 
                            rvChatImportant.getA 
                            dapter().getItemCount() - 1;
                            scrollTo = scrollTo >= 0 ? 
                            scrollTo : 0;                             
                            rvChatImportant. 
                            scrollToPosition(scrollTo);
                        }
                    }, 10);
                }
            }
        });
    }
}
Continue ReadingControl flow of SUSI AI on Android and database management using Realm

Deploying SUSI-Server on Digital Ocean

Create and Setup a Droplet

To create your first Droplet first login to your account. If you are using Github student developer pack then you will get $50 as Digital Ocean Credits.

The create button will be right there on the first page, click on “Create Droplet”
Then a page will open asking you some configurations of a server.

You are free to choose any Distribution and choose a size. I chose Ubuntu and $20/mo plan.
Now add information about the Data Center you want to use.

Add your SSH Keys. These keys will be used to authenticate you on remote login to your server.

Click Create. In a few seconds your droplet should be up.

Navigate to Droplets page and see that your droplet is live there. You should see your droplet name and IP there.

SSH to your server and Deploy SUSI

In order to connect to a remote Linux server via SSH, you must have following:

  • User name: The remote user to log in as. The default admin user, or Superuser, on most Linux servers is root
  • Password and/or SSH Key: The password that is used to authenticate the user that you are logging in as. If you added a public SSH key to your droplet when you created it, you must have the private SSH key of the key pair (and passphrase, if it has one)
  • Server IP address: This is the address that uniquely identifies your server on the Internet.

Open Terminal and run

ssh root@SERVER_IP_ADDRESS

You should be able to login to your server.

Now to run SUSI Server

(Step 1 to 10 to setup java and gradle, skip if you have done it before)

Visit SERVER_IP_ADDRESS:4000 and see that SUSI Server is running there.

To stop the server use:

bin/stop.sh

Continue ReadingDeploying SUSI-Server on Digital Ocean

Detecting and Fixing Memory leaks in Susi Android App

In the fast development of the Susi App, somehow developers missed out some memory leaks in the app. It is a very common mistake that developers do. Most new android developers don’t know much about Memory leaks and how to fix them. Memory leaks makes the app slower and causes crashes due to OutOfMemoryException. To make the susi app more efficient, it is advised to look out for these memory leaks and fix them. This post will focus on teaching developers who’ll be contributing in Susi android App or any other android app about the memory leaks, detecting them and fixing them.

What is a memory leak?

Android system manages memory allocation to run the apps efficiently. When memory runs short, it triggers Garbage Collector (GC) which cleans up the objects which are no longer useful, clearing up memory for other useful objects. But, suppose a case when a non-useful object is referenced from a useful object. In that case Garbage Collector would mark the non-useful object as useful and thus won’t be able to remove it, causing a Memory Leak.

Now, when a memory leak occurs, the app demands for memory from the android system but the android system can only give a certain amount of memory and after that point it will refuse to give more memory and thus causing a OutOfMemoryException and crashing the app. Even if sometime due to memory leaks, the app doesn’t crash but it surely will slow down and skip frames.

Now, few other questions arises, like “How to detect these leaks?” , “What causes these leaks ?” and “How can we fix these?” Let’s cover these one by one.

Detecting Memory Leaks

You can detect memory leaks in android app in two ways :

  1. Using Android Studio
  2. Using Leak Canary

In this post I’ll be describing the way to use Leak Canary to detect Memory Leaks. If you want to know about the way to use android studio to detect leaks, check out this link .

Using Leak Canary for Memory Leak detection :

Leak Canary is a very handy tool when it comes to detecting memory leaks. It shows the memory leak in an another app in the mobile itself. Just add these lines under the dependencies in build.gradle file.

debugCompile ‘com.squareup.leakcanary:leakcanary-android:1.5.1’
releaseCompile ‘com.squareup.leakcanary:leakcanary-android-no-op:1.5.1’
testCompile ‘com.squareup.leakcanary:leakcanary-android-no-op:1.5.1’

And this code here in the MainApplication.java file.

if (LeakCanary.isInAnalyzerProcess(this)) {
// This process is dedicated to LeakCanary for heap analysis.

// You should not init your app in this process.

return
;
}
LeakCanary.install(this);
// Normal app init code…

You are good to go. Now just run the app and if there is a memory leak you will get something like this. It dumps the memory in .hprof file and displays it in another app.

      

Causes of Memory Leaks

There are many causes of memory leaks. I will list a few top of my head but there can be more.

  1. Static Activities and Views : Defining a static variable inside the class definition of the Activity and then setting it to the running instance of that Activity. If this reference is not cleared before the Activity’s lifecycle completes, the Activity will be leaked.
  2. Listeners : When you register a listener, it is advised to unregister it in onDestroy() method to prevent memory leaks. It is not that prominent but may cause memory leaks.
  3. Inner Classes : If you create an instance of Inner Class and maintain a static reference to it, there is a chance of memory leak.
  4. Anonymous Classes : A leak can occur if you declare and instantiate an AsyncTask anonymously inside your Activity. If it continues to perform background work after the Activity has been destroyed, the reference to the Activity will persist and it won’t be garbage collected until after the background task completes.
  5. Handlers and Threads : The very same principle applies to background tasks declared anonymously by a Runnable object and queued up for execution by a Handler object.

Preventing and Fixing Memory Leaks

So, now you know what are the causes of these memory leaks. You just have to be a little more careful while implementing these. Here are some more tips to prevent or fix memory leaks :

  1. Be extra careful when dealing with Inner classes and Anonymous classes. Make them static wherever possible. Use a static inner class with a WeakReference to the outer class if that helps.
  2. Be very careful with a static variable in your activity class because it can reference your activity and cause leak. Be sure to remove the reference in onDestroy().
  3. Unregister all listeners in onDestroy() method.
  4. Always terminate worker threads you initiated on Activity onDestroy().
  5. Make sure that your allocated resources are all collected as expected. Do not always rely on Garbage Collector.
  6. Try using the context-application instead of a context-activity.

Conclusion

So, now if you want to contribute in Susi Android App and implement a feature in it, you can just check if there is a memory leak due to your implementation and fix it for better performance of the app. Also, if you find any other memory leak in the app, do report it on the issue tracker, fix it and make the Susi Android App more efficient.

Happy Coding!

Continue ReadingDetecting and Fixing Memory leaks in Susi Android App

How to teach SUSI skills calling an External API

SUSI is an intelligent  personal assistant. SUSI can learn skills to understand and respond to user queries better. A skill is taught using rules. Writing rules is an easy task and one doesn’t need any programming background too. Anyone can start contributing. Check out these tutorials and do watch this video to get started and start teaching susi.

SUSI can be taught to call external API’s to answer user queries.

While writing skills we first mention string patterns to match the user’s query and then tell SUSI what to do with the matched pattern. The pattern matching is similar to regular expressions and we can also retrieve the matched parameters using $<parameter number>$ notation.

Example :

My name is *
Hi $1$!

When the user inputs “My name is Uday” , it is matched with “My name is *” and “Uday” is stored in $1$. So the output given is “Hi Uday!”.

SUSI can call an external API to reply to user query. An API endpoint or url when called must return a JSON or JSONP response for SUSI to be able to parse the response and retrieve the answer.

Rule Format for a skill calling an external API

The rule format for calling an external API is :

<regular expression for pattern matching>
!console: <return answer using $object$ or $required_key$>
{
  “url”:<API endpoint or url>,
  “path”:$.<key in the API response to find the answer>,
}
eol
  • Url is the API endpoint to be called which returns a JSON or JSONP response.
    The parameters to the url if any can be added using $$ notation.
  • Path is used to help susi know where to look for the answer in the returned response.
    If the path points to a root element, then the answer is stored in $object$, otherwise we can query $key$ to get the answer which is a value to the key under the path.
  • eol or end of line indicates the end of the rule.

Understanding the Path Attribute

Let us understand the Path attribute better through some test cases.

In each of the test cases we discuss what the path should be and how to retrieve the answer for a given required answer from the json response of an API.

  1. API response in json :

    {
      “Key1” : “Value1”
    }
    

Required answer : Value1
Path : “$.Key1    =>   Retrieve Answer:  $object$

 

  1. API response in json :
{
  “Key1” : [{“Key11” : “Value11”}]
}

Required answer : Value11
Path : $.Key1[0]   =>  Retrieve Answer: $Key11$
Path : $.Key1[0].Key11   => Retrieve Answer: $object$

 

  1. API response in json :
{
  “Key1” : {“Key11” : “Value11”}
}


Required answer : Value11
Path : $.Key1  => Retrieve Answer:  $Key11$
Path : $.Key1.Key11  => Retrieve Answer: $object$

 

  1. API response in json :
{
  “Key1” : {
           “Key11” : “Value11”,
           “Key12” : “Value12”
         }
}

Required answer : Value11 , Value12
Path : $.Key1  => Retrieve Answer:  $Key11$ , $Key12$

Where to write these rules?

Now, since we know how to write rules let’s see where to write them.

We use etherpads to write and test rules and once we finish testing our rule we can push those rules to the repo.

Steps to open, write and test rules:

  1. Open a new etherpad with a desired name <etherpad name> at http://dream.susi.ai/
  2. Write your skills code in the etherpad following the code format explained above.
  3. Now, to test your skill let’s chat with susi. Start a conversation with susi at http://susi.ai/chat to test your skills.
  4. Load your skills by typing dream <etherpad name> and wait for a response saying dreaming enabled for <etherpad name>
  5. Test your skill and follow step 4 every time you make changes to the code in your etherpad.
  6. After you are done testing, type stop dreaming and if you are satisfied with your skill do send a PR to help susi learn.

Examples

Let us try an example to understand this better.

1. Plot of a TV Show

Tvmaze is an open  TV API that provides information about tv shows. Let us write a rule to know the plot of a tv show. We can find many such APIs. Check out this link listing few of them.

  1.  Open an etherpad at http://dream.susi.ai/ named tvshowplot.
  2.   Enter the code to query plot of a TV show in the etherpad at                           http://dream.susi.ai/p/tvshowplot
* plot of *|* summary of *
!console:$object$
{
  "url":"http://api.tvmaze.com/singlesearch/shows?q=$2$",
  "path":"$.summary"
}
eol
  1. Now lets test our skill by starting a conversation with susi at http://susi.ai/chat.
  • User Query: dream tvshowplot
    Response:  dreaming enabled  for tvshowplot
  • User Query: what is the plot of legion
    Response: Legion introduces the story of David Haller: Since he was a teenager, David has struggled with mental illness. Diagnosed as schizophrenic, David has been in and out of psychiatric hospitals for years. But after a strange encounter with a fellow patient, he’s confronted with the possibility that the voices he hears and the visions he sees might be real. He’s based on the Marvel comics character Legion, the son of X-Men founder Charles Xavier (played by Patrick Stewart and James McAvoy in the films), first introduced in 1985.

Intermediate Processing:

Pattern Matching : $1$ = “what is the” ; $2$ = “legion”

Url : http://api.tvmaze.com/singlesearch/shows?q=legion

API response:

{
  "id": 6393,
  "url": "http:\/\/www.tvmaze.com\/shows\/6393\/legion",
  "name": "Legion",
  "type": "Scripted",
  "language": "English",
  "genres": [
             "Drama",
             "Action",
             "Science-Fiction"
            ],
  "summary": "<p><strong>Legion<\/strong> introduces the story of David Haller: Since he was a teenager, David has struggled with mental illness. Diagnosed as schizophrenic, David has been in and out of psychiatric hospitals for years. But after a strange encounter with a fellow patient, he's confronted with the possibility that the voices he hears and the visions he sees might be real. He's based on the Marvel comics character Legion, the son of X-Men founder Charles Xavier (played by Patrick Stewart and James McAvoy in the films), first introduced in 1985.<\/p>",
  "updated": 1491955072,
}

Note: The API response has been trimmed to show only the relevant content.

Path : $.summary

Retrieving Answer: so our required answer in the api response is under the key summary and is retrieved using $object$ since it is a root element.

 

Screenshots:

2. Cooking Recipes

Let us try it out with another API.
Recipepuppy is an cooking recipe API where users can query various recipes.

  1.  Open a etherpad at http://dream.susi.ai/ named recipe.
  2.   Enter the code to query a recipe in the etherpad at  http://dream.susi.ai/p/recipe
#Gives recipes and links to cook a dish
* cook *
!console:<p>To cook  <strong>$title$</strong> : <br>The ingridients required are: $ingredients$. <br> For instruction to prepare the dish $href$ </p>
{
  "url":"http://www.recipepuppy.com/api/?q=$2$",
  "path":"$.results"
}
eol
  1. Now lets test our skill by starting a conversation with susi at http://susi.ai/chat.
  • User Query: dream recipe
    Response:  dreaming enabled  for recipe
  • User Query: how to cook chicken biryani
    Response: To cook Chicken Biryani Recipe :
    The ingridients required are: chicken, seeds, chicken broth, rice, butter, peas, garlic, red onions, cardamom, curry paste, olive oil, tomato, coriander, cumin, brown sugar, tumeric.
    For instruction to prepare the dish Click Here!

Intermediate Processing:

Pattern Matching : $1$ = “how to” ; $2$ = “chicken biryani”

Url : http://www.recipepuppy.com/api/?q=chicken biryani

API response:

{
  "title": "Recipe Puppy",
  "version": 0.1,
  "href": "http:\/\/www.recipepuppy.com\/",
  "results": [
              {
                "title": "Chicken Biryani Recipe",
                "href": "http:\/\/www.grouprecipes.com\/53040\/chicken-biryani.html",
                "ingredients": "chicken, seeds, chicken broth, rice, butter, peas, garlic, red onions, cardamom, curry paste, olive oil, tomato, coriander, cumin, brown sugar, tumeric",
                "thumbnail": "http:\/\/img.recipepuppy.com\/413822.jpg"
             },
           ]
}

Note: The API response has been trimmed to show only the relevant content.

Path : $.results[0]

Retrieving Answer: so our required answer in the api response is under the key results and since it’s an array we are using the first element of the array and since the element is a dictionary too we use its keys correspondingly to answer. The $href$ is rendered as “Click Here” hyperlinked to the actual url.

 

Screenshots:

 

We have successfully taught susi a skill which tells users about the plot of a tv show and a skill to query recipes.
Cheers!
Following similar procedure, we can make use of other APIs and teach susi several new skills.

Resources
Continue ReadingHow to teach SUSI skills calling an External API

Deploying Susi Server on Google Cloud with Kubernetes

Susi (acronym for Scientific User Support Intelligence) is an advanced AI made by people at FOSSASIA. It is an AI made by the people and for the people.
Susi is an Open Source Project under LGPL Licence.

SUSI.AI already has many Skills and anyone can add new skills through simple console rules.

If you want to participate in the development of the SUSI server you can start by learning to deploy it on a cloud system like Google Cloud.

This way whenever you make a change to Susi Server, you can test it out on various Susi Apps instantly.

Google Cloud with Kubernetes provide this ability. Let’s dig deep into what is Google Cloud Platform and Kubernetes.

What is Google Cloud Platform ?

Google Cloud Platform lets you build and host applications and websites, store data, and analyze data on Google’s scalable infrastructure.
Google Cloud Platform (at the time of writing this article) also provides free credits worth $300 for 1 year for testing out the Platform and test your applications.

What is Kubernetes ?

Kubernetes is an open-source system for automatic deployment, management and scaling of containerized applications. It makes it easy to roll out updates to your application with simple commands from your development machine and scale horizontally easily by adding more clusters as demand increase.

Deploying Susi Server on Kubernetes

Deploying Susi Server on Kubernetes is a fairly easy task. Follow up the steps to get it running.

Create a Google Cloud Account

Sign up for a Google Cloud Account (https://cloud.google.com/free-trial/) and get 300$ credits for initial use.

Create a New Project

After successful sign up, create a new project on Google Cloud Console.
Let’s name it Susi-Kubernetes . 

You will be provided a ProjectID. Remember it for further reference.

Install Google Cloud SDK and kubectl

Go to https://cloud.google.com/sdk/ and see instructions to setup Google Cloud SDK on your respective OS.

After Google Cloud SDK install, run

gcloud components install kubectl

This will install kubectl for interacting with Kubernetes.

Login and setup project

  1. Login to your Google Cloud Account using
$ gcloud auth login

2. List all the projects using

$ gcloud config list project
[core]
project = <PROJECT_ID>

3. Select your project

$ gcloud config set project <PROJECT_ID>

4. Install JDK8 for susi_server setup and set it as default.

5. Clone your fork of the Susi Server Repository

$ git clone https://github.com/<your_username>/susi_server.git
$ cd susi_server/

6. Build project and run Susi Server locally

$ ./gradlew build
$ bin/start.sh

Susi server must have been started started and web interface is accessible on http://localhost:4000

Install Docker and build Docker image for Susi

  1. Install Docker.
    Debian and derivatives:  sudo apt install docker
    Arch Linux:   sudo pacman -S docker 
  2. Build Docker Image for Susi
    $ docker build -t gcr.io/<Project_id>/susi:v1 .
  3. Push Image to Google Container Registry private to your project.
$ gcloud docker -- push gcr.io/<Project_id>/susi:v1

Create Cluster and Deploy your Susi Server there

  1. Create Cluster. You may specify different zone, number of nodes and machine type depending upon requirement.
    $ gcloud container clusters create <Cluster-Name> --num-nodes 2 --machine-type n1-standard-1 --zone us-central1-c
  2. Run your deployment. You may specify any name for deployment.
    $ kubectl run <deployment_name> --image=gcr.io/<Project_id>/susi:v1 --port=80
    $ kubectl get deployments
    $ kubectl expose deployment susi --type=LoadBalancer
  3. Check your deployment and get Public IP for Access.
    $ kubectl get services
    NAME         CLUSTER-IP     EXTERNAL-IP     PORT(S)       AGE
    kubernetes   10.3.240.1     <none>          443/TCP        1d
    susi         10.3.241.145   <PUBLIC_IP>     80:31155/TCP   1d
  4. Go to provided public IP to check, if Susi Server is running.

Congratulations, you successfully setup Susi Server on Google Cloud with Kubernetes.

Updating the deployment

Next step is to update deployment when you wish to roll out changes. To do so.

Build Docker Image and Push it to Google Container Registry

$ docker build -t gcr.io/<Project_Id>/susi:v2 .
$ gcloud docker -- push gcr.io/<Project_Id>/susi:v2

Update Deployment Image with Kubernetes

$ kubectl set image deployment/<Deployment_Name> \
  <Deployment_Name>=gcr.io/<Project_id>/susi:v2
deployment "<Deployment_Name>" image updated

Go to public ip to see the changes.

That’s it. Now, you have fully running Susi Server on your own Google Cloud Cluster using Kubernetes.

Continue ReadingDeploying Susi Server on Google Cloud with Kubernetes

Susi AI Skill Development

What is Susi?

Susi is an open source intelligent personal assistant which has the capability to learn and respond better to queries. It is also capable of making to-do lists, setting alarms, providing weather and traffic info all in real time. Susi responds based on skills.

What is a skill? How do we teach a skill?

A skill is a piece of code which performs a set of actions in order to respond to the user’s query. These skills are based on pattern matching which help them mapping the user’s query to a specific skill and responding accordingly. Teaching a skill to Susi is surprisingly very easy to implement. One can take a look at the Susi Skill Development Tutorial and a video workshop by Michael Christen.

I will try to give a basic idea on how to create a skill, it’s basic structure and some of the skills I developed in the first week.

Prepare to create a skill:

  • Head over to http://dream.susi.ai
  • Create a etherpad with some relevant name
  • Delete all text currently present in there
  • Start writing your skill

Adding to this, for testing a skill one can head over to Susi Web Chat Interface.

Basic Structure for calling an API:

<Regular expression to be matched here>

!console:<response given to the user>
 {
 "url":"<API endpoint>",
 "path":"<Json path here>"
 }
 eol

So, let me explain this line by line.

  1. The regular expression is the one to which the user’s query is matched first.
  2. The console is meant to output the actual response the user sees as response.
  3. In place of the “url”, the API endpoint is passed in.
  4. “path” here specifies how we traverse through the response Json or Jsonp to get the object, starts with “$.”.
  5. At last, “eol” which is the end-of-line marks the end of a skill.

Let’s take an example for better understanding of this:

random gif
!console: $url$
{
    "url" : "http://api.giphy.com/v1/gifs/trending?api_key=dc6zaTOxFJmzC",
    "path" : "$.data[0].images.fixed_height"
}
eol 

 

This skill responds with a link to a random gif.

Steps involved:

  1. Match the string “random gif” with the user’s query.
  2. On successful match, make an API call to the API endpoint specified in “url”
  3. On response, extract the object at the specified path in the json under “path”
  4. Respond to the user with the “url” key’s value which would here be an URL of a GIF.

Let’s try it out on Susi Web Chat. For this, you will first have to load your skill using the dream command followed by etherpad name: dream <etherpad name>. And then you can start testing your skill.

So, we queried “random gif” and we got a response “Click Here!”. The complete URL didn’t show up because all the URLs are currently parsed and a hyperlink for each is created. So try clicking on it to find a GIF.

 

Now, let’s look at one more skill I developed during this period.

# Returns the name of the president of a country

 president of *|who is the president of *| president *
 !console:$plaintext$
 {      "url":"https://api.wolframalpha.com/v2/query?input=president+$1$&output=JSON&appid=9WA6XR-26EWTGEVTE&includepodid=Result",
   "path" : "$.queryresult.pods[0].subpods[0]"
 }
 eol

 

Let’s understand this step by step:

  1. We have here “president of *|who is the president of *| president *”, which means the user’s query matches with anyone of the following because of the use of pipe symbol “|”. The “*” here replaces a word or a list of words, which can be accessed like: “${index}$”  where index is replaced by the position of the “*” in the expression starting from 1.
  2. Now we have something new in the URL. See that  $1$  inside the URL? On runtime, that is replaced with the content of the “*” variable. So if a user puts in query like: “president of usa”, “usa” is mapped to $1$ and is replaced in the URL and appropriate API request is made.
  3. Then the path is traversed in the json response and the value of the “plaintext” key is used to respond to the user.

 

It’s now time to try it out on Susi Web Chat.

So, we got our desired response here, i.e., the name of the president of usa.

Continue ReadingSusi AI Skill Development