Map Support for SUSI Webchat

SUSI chat client supports map tiles now for queries related to location. SUSI responds with an interactive internal map tile with the location pointed by a marker. It also provides you with a link to open street maps where you can get the whole view of the location using the zooming options provided and also gives the population count for that location.

Lets visit SUSI WebChat and try it out.

Query : Where is london
Response :

Implementation:

How do we know that a map tile is to be rendered?
The actions in the API response tell the client what to render. The client loops through the actions array and renders the response for each action accordingly.

"actions": [
  {
    "type": "answer",
    "expression": "City of London is a place with a population of 7556900.             Here is a map: https://www.openstreetmap.org/#map=13/51.51279067225417/-0.09184009399817228"
  },
  {
    "type": "anchor",
    "link":    "https://www.openstreetmap.org/#map=13/51.51279067225417/-0.09184009399817228",
    "text": "Link to Openstreetmap: City of London"
  },
  {
    "type": "map",
    "latitude": "51.51279067225417",
    "longitude": "-0.09184009399817228",
    "zoom": "13"
  }
]

Note: The API response has been trimmed to show only the relevant content.

The first action element is of type answer so the client renders the text response, ‘City of London is a place with a population of 7556900. Here is a map: https://www.openstreetmap.org/#map=13/51.51279067225417/-0.09184009399817228

The second action element is of type anchor with the text to display and the link to hyperlink specified by the text and link attributes, so the client renders the text `Link to Openstreetmap: City of London`, hyperlinked to “https://www.openstreetmap.org/#map=13/51.51279067225417/-0.09184009399817228”.

Finally, the third action element is of type map. Latitude, Longitude and zoom level information are also  specified using latitude, longitude and zoom attributes. The client renders a map using these attributes.

I used react-leafletmodule to render the interactive map tiles.

To integrate it into our project and set the required style for the map tiles, we need to load Leaflet’s CSS style sheet and we also need to include height and width for the map component. 

<link rel="stylesheet"  href="http://cdn.leafletjs.com/leaflet/v0.7.7/leaflet.css" />
.leaflet-container {
  height: 150px;
  width: 80%;
  margin: 0 auto;
}
case 'map': {

  let lat = parseFloat(data.answers[0].actions[index].latitude);
  let lng = parseFloat(data.answers[0].actions[index].longitude);
  let zoom = parseFloat(data.answers[0].actions[index].zoom);
  let mymap = drawMap(lat,lng,zoom);

  listItems.push(
    <li className='message-list-item' key={action+index}>
      <section className={messageContainerClasses}>
        {mymap}
        <p className='message-time'>
          {message.date.toLocaleTimeString()}
        </p>;
      </section>
    </li>
  );

  break;
}
import { divIcon } from 'leaflet';
import { Map, Marker, Popup, TileLayer } from 'react-leaflet';


// Draw a Map

function drawMap(lat,lng,zoom){

  let position = [lat, lng];

  const icon = divIcon({
    className: 'map-marker-icon',
    iconSize: [35, 35]
    });

  const map = (
    <Map center={position} zoom={zoom}>
      <TileLayer
      attribution=''
      url='http://{s}.tile.osm.org/{z}/{x}/{y}.png'
      />
      <ExtendedMarker position={position} icon={icon}>
        <Popup>
          <span><strong>Hello!</strong> <br/> I am here.</span>
        </Popup>
      </ExtendedMarker>
    </Map>
  );

return map;

}

Here, I used a custom marker icon because the default icon provided by leaflet had an issue and was not being rendered. I used divIcon from leaflet to create a custom map marker icon.

When the map tile is rendered, we see a Popup message at the marker. The extended marker class is used to keep the Popup open initially.

class ExtendedMarker extends Marker {
  componentDidMount() {
    super.componentDidMount();
    this.leafletElement.openPopup();
  }
}


The function drawMap returns a Map tile component which is rendered and we have our interactive map!

Resources
Continue ReadingMap Support for SUSI Webchat

Hyperlinking Support for SUSI Webchat

SUSI responses can contain links or email ids. Whenever we want to access those links or send a mail to those email ids, it is very inconvenient for the user to manually copy the link and check out the contents, which is a very bad UX.

I used a module called ‘react-linkify’ to address this issue.
React-linkify’ is a React component to parse links (urls, emails, etc.) in text into clickable links.

Usage:

<Linkify>{text to linkify}</Linkify>

Any link that appears inside the linkify component will be hyperlinked and is made clickable. It uses regular expressions and pattern matching to detect URLs and mail ids and are made clickable such that clicking on URLs opens the link in a new window and clicking a mail id opens “mailto:” .

Code:

export const parseAndReplace = (text) => {return <Linkify properties={{target:"_blank"}}>{text}</Linkify>;}

Lets visit SUSI WebChat and try it out.

Query: search internet

Response: Internet The global system of interconnected computer networks that use the Internet protocol suite to… https://duckduckgo.com/Internet

The link has been parsed from the response text and has been successfully hyperlinked. Clicking the links opens the respective URL in a new window.

Resources
Continue ReadingHyperlinking Support for SUSI Webchat

Giving a Voice to Susi

Susi AI already has various apps and is available as a chatbot in various messaging platforms. We are going a step forward to make an SDK available for Susi that can be integrated on any Hardware Device (say speakers, toys, your bicycle etc – possibilities are endless )

One of the problem that I encountered while making a Prototype for the same is selecting an appropriate Text to Speech (TTS)  Engine.

It was a challenge, since on platforms like Android and iOS , you may utilize TTS engines bundled with Platform easily via a platform specific API, which are well optimized and give good performance.  The same was difficult on a hardware device that can run only Linux with no TTS provided by default.

Thus, I explored some possibilities

eSpeak TTS: eSpeak TTS (http://espeak.sourceforge.net/) was the first option considered for the task.
eSpeak is a compact open source software speech synthesizer for English and other languages, for Linux and Windows.

The major advantage of eSpeak is its small size (2MB) and small memory footprint which is advantageous in Low Memory Hardware like Orange Pi Zero or Raspberry Pi Zero.


Setting up eSpeak was easy but with its advantages , there were some drawbacks too.

  • The voice synthesis was quite robotic.
  • Very few voices were available.

Festival TTS: Festival offers a general framework for building speech synthesis systems as well as including examples of various modules. As a whole it offers full text to speech through a number APIs: from shell level, though a Scheme command interpreter, as a C++ library, from Java, and an Emacs interface.


Festival is free software. Festival and the speech tools are distributed under an X11-type licence allowing unrestricted commercial and non-commercial use alike.

Installing Festival:

On Arch Linux , it was pretty straight forward.

sudo pacman -S festival

There is a full wiki dedicated to it. ( https://wiki.archlinux.org/index.php/Festival )

Testing Festival

Festival has an interpreter to test it out. It can be invoked using
You may test out a TTS output using:

festival> (SayText "Hi!! I am Susi")

But the default sound in festival is still robotic and male. You don’t want your Personal Assistant to scare you out when you speak to her.

Thus, I searched on what are the best female voices available for Festival.

After looking at a discussion on the thread, https://ubuntuforums.org/showthread.php?t=751169 ,

I found that CMU-Arctic and HTS are some of the best voice sets for  Festival.

In Arch Linux, additional voice packs, are supplied in two additional packages,

festival-us and festival-english

Installation is straightforward:

sudo pacman -S festival-us festival-english

Now, on festival REPL , we can test out our new voices.

To see all available voices

festival> (voice.list)
(rab_diphone kal_diphone cmu_us_rms_cg cmu_us_awb_cg cmu_us_slt_cg)

Testing out a voice

festival> (voice_cmu_us_awb_cg)
cmu_us_awb_cg
festival> (SayText "Hi!! I am Susi")

This way after testing out all voices, with many different phrases. cmu_us_slt_cg  felt like an appropriate voice.

Setting Voice as Default

Voice may be set as default by adding following line to .festivalrc

 (set! voice_default voice_cmu_us_slt_cg)

Now festival will be invoked with this voice as default.

You may read a file using festival using

$ festival --tts <filename>

Calling Festival from Python

This was accomplished by first writing to a file, and then calling a subprocess to output speech using festival tts.

def speak(text):
    filename = '.response'
    file=open(filename,'w')
    file.write(text)
    file.close()
    # Call festival tts to reply the response by Susi
    subprocess.call('festival --tts '+filename, shell=True)

 

So, festival works pretty well on a moderately powerful machine, but while trying it on Raspberry Pi with a custom voice, there was noticeable amount of time delay for synthesis, so I searched for more alternatives.

Flite TTS: CMU Flite (festival-lite) is a small, fast run-time open source text to speech synthesis engine developed at CMU and primarily designed for small embedded machines and/or large servers. Flite is designed as an alternative text to speech synthesis engine to Festival for voices built using the FestVox suite of voice building tools.
Flite gives a noticeable speech improvement. It gives a Festival like output for 1/10th the amount taken by Festival.

Installing Flite TTS

On Raspberry Pi (Raspbian) , the version in Raspbian Jessie Repository is Flite 1.4 which does not support using an external speech file, so we need to compile and install from sources.

$ wget http://www.festvox.org/flite/packed/latest/flite-2.0.0-release.tar.bz2
$ tar xf flite-2.0.0-release.tar.bz2
$ cd flite-2.0.0-release/
$ ./configure
$ make
$ sudo make install

Now, you may download flitevox file for voice you wish to use from

http://www.festvox.org/flite/packed/latest/voices/

It can be invoked using

$ flite -voice file://<flitevox_file_path> -f <filepath-to-read>

You may save output audio stream to a file using

$ flite -voice file://<flitevox_file_path> -f <filepath-to-read> -o output.wav

Now, you can playback the audio. This can be invoked from python using.

import os

def speak_flite_tts(text):
    filename = '.response'
    file = open(filename, 'w')
    file.write(text)
    file.close()
    # Call flite tts to reply the response by Susi
    flite_speech_file = 'cmu_us_slt.flitevox'
    print('flite -voice file://{0} -f {1}'.format(flite_speech_file, filename))
    os.system('flite -v -voice file://{0} -f {1} -o output.wav'.format(flite_speech_file, filename))
    os.system('aplay output.wav')

 

In this way, we added offline Text to Speech Support of Susi Hardware SDK.

Continue ReadingGiving a Voice to Susi

Implementing a chatbot using the SUSI.AI API

SUSI AI is an intelligent Open Source personal assistant. It is a server application which is able to interact with humans as a personal assistant. The first step in implementing a bot using SUSI AI is to specify the pathway for query response from SUSI AI server.

The steps mentioned below provide a step-by-step guide to establish communication with SUSI AI server:

    1. Given below is HTML code that demonstrates how to connect with SUSI API through an AJAX call. To put this file on a Node Js server, see Step 2.  To view the response of this call, follow Step 4.

      <!DOCTYPE html>
      <body>
      <h1>My Header</h1>
      <p>My paragraph.</p>
      //Script with source here 
      //Script to be written here 
      </body>
      </html>
      

      In above code add scripts given below and end each script with closing tag </script>. In the second script we are calling SUSI API with hello query and showing data that we are receiving through call on console.

      <script src="https://ajax.googleapis.com/ajax/libs/jquery/3.2.1/jquery.min.js">
    2. <script>
      $(function (){ 
          $.ajax({ 
              dataType: 'jsonp', 
              type:'GET', url: 'http://api.susi.ai/susi/chat.json? timezoneOffset=-300&q=hello', 
               success: function(data){ 
                   console.log('success', data); 
               } 
          }); 
      });
    3. Code below is in node js to setup localhost and getting the same above result on browser. Below is Node Js code to setup a server at localhost for the above created HTML file.

      var http = require('http');
       var fs = require('fs');
       http.createServer(function (req, res) {
        fs.readFile('YOURFILENAME.html', function(err, data) {
          res.writeHead(200, {'Content-Type': 'text/html'});
          res.write(data);
          res.end();
        });
       }).listen(9000);
    4. We will get following response by running this Node js code and checking results on http://localhost:9000/ To run this code install Node Js and write “node filename.js” in command line.
    5. You can open above window by right clicking on page and selecting Inspect. Go to the Network option and select the relevant api call from left section of Inspect window.

We have successfully got response from SUSI API and now we can use this response for building bots for receiving replies for user.

Continue ReadingImplementing a chatbot using the SUSI.AI API

Hotword Detection in Susi Android

Hotword detection is one of the coolest features in Android. Voice control has emerged as a popular method for interacting with smartphones and wearable devices. It allows the user to interact with the app without even touching the device in a much intuitive way. At the same time it’s difficult to implement them in the Android app and requires use of large number of resources. Let us dive deeper into this topic.

So, Firstly What is a Hotword?

Hotword is generally a phrase or a word that can be used to initiate a particular task in the app. The user needs to speak the phrase or the hotword which on detection makes a callback that can be used to carry out a particular task in the app.

Examples of hotword include “Ok Google” which is used to initiate Google assistant in the Android mobile phones.

Why is it difficult to implement hotword detection?

To enable hotword detection is a cumbersome task. There are different problems associated with it including:-

  1. Change in the accent of the speaker.
  2. Continuous task of recognizing the voice and then matching it with the hotword.
  3. The features needs to be run as a service in android which causes the drain of battery.
  4. Also it is a memory intensive task at the same time.

Let’s us look at the present techniques available for the hotword detection.

  1. The Android provides a class called as AlwaysOnHotwordDetector. This class can be used to detect the keywords inside the activity of an Android app. For more details you can visit this link.

The implementation goes like this

/**

* @param text The keyphrase text to get the detector for.

* @param locale The java locale for the detector.

* @param callback A non-null Callback for receiving the recognition events.

* @param voiceInteractionService The current voice interaction service.

* @param modelManagementService A service that allows management of sound models.

*

* @hide

*/


public AlwaysOnHotwordDetector(String text, Locale locale, Callback callback, KeyphraseEnrollmentInfo keyphraseEnrollmentInfo, IVoiceInteractionService voiceInteractionService, IVoiceInteractionManagerService modelManagementService) {

  mText = text;

  mLocale = locale;

  mKeyphraseEnrollmentInfo = keyphraseEnrollmentInfo;

  mKeyphraseMetadata = mKeyphraseEnrollmentInfo.getKeyphraseMetadata(text, locale);

  mExternalCallback = callback;

  mHandler = new MyHandler();

  mInternalCallback = new SoundTriggerListener(mHandler);

  mVoiceInteractionService = voiceInteractionService;

  mModelManagementService = modelManagementService;

  new RefreshAvailabiltyTask().execute();



}

2.  We can also detect hotword using Pocketsphinx library in Android. The library acts as a service in the Android app. It is quite efficient when it comes to battery consumption. The implementation goes like this:-    

recognizer = defaultSetup()
   .setAcousticModel(new File(assetsDir, "en-us-ptm"))
   .setDictionary(new File(assetsDir, 
"cmudict-en-us.dict"))
   .getRecognizer();
recognizer.addListener(this);



// Create keyword-activation search.
recognizer.addKeyphraseSearch(KWS_SEARCH, KEYPHRASE);

// Create grammar-based searches.
File menuGrammar = new File(assetsDir, "menu.gram");
recognizer.addGrammarSearch(MENU_SEARCH, menuGrammar);

// Next search for digits
File digitsGrammar = new File(assetsDir, "digits.gram");
recognizer.addGrammarSearch(DIGITS_SEARCH, digitsGrammar);

// Create language model search.
File languageModel = new File(assetsDir, "weather.dmp");
recognizer.addNgramSearch(FORECAST_SEARCH, languageModel);


recognizer.startListening(searchName);

Now Lets us look at how we are implementing it in Susi Android:

In Susi Android, we have used CMU’s Pocketsphinx library for the hotword detection. We are using “Hi Susi” as the hotword for the detection.

private SpeechRecognizer recognizer;

/* Keyword we are looking for to activate menu */

private static final String KEYPHRASE = "hi susi";

/* Named searches allow to quickly reconfigure the decoder */

private static final String KWS_SEARCH = "hi susi";



The function setupRecognizer is used to initialize the Recognizer and detect the keyphrase



private void setupRecognizer(File assetsDir) throws IOException {

// The recognizer can be configured to perform multiple searches

// of different kind and switch between them

recognizer = SpeechRecognizerSetup.defaultSetup()

.setAcousticModel(new File(assetsDir, "en-us-ptm"))

.setDictionary(new File(assetsDir, "cmudict-en-us.dict"))

.setRawLogDir(assetsDir)

.getRecognizer();

recognizer.addListener(this);

recognizer.addKeyphraseSearch(KWS_SEARCH, KEYPHRASE);

There is still some need of improvement towards this approach in Susi app as the success rate of detection of keyword is not very high. We can overcome this by training the model or having a set of strings similar to that of “Hi Susi” such as “Hii Susi” or “Hi Sushi”. This will increase the rate of detection.  

Continue ReadingHotword Detection in Susi Android

Deploying SUSI.AI with Docker

Docker is much more efficient than VM in allocating shared resources between various containers as shown in figure. To deploy SUSI we need to create docker container. There are two ways to build it. First way is fork the SUSI project in github. Then you can signup in dockerhub and create autobuild docker container. The second way is to manually build docker file from command prompt of your computer. The following instructions needs to be executed in cloud shell or linux machine.

sudo apt-get update
sudo apt-get upgrade
sudo apt-get -y install docker.io
sudo fallocate -l 4G /swapfile
sudo chmod 600 /swapfile
sudo mkswap /swapfile
sudo swapon /swapfile
sudo docker build https://github.com/fossasia/susi_server.git


The first three commands install docker software to the machine. Next three lines give required permissions and execution abilities to docker. Final command builds docker container from github. Thus, We have successfully made docker container. We can deploy it on the cloud by using following command.

sudo docker run -d -p 80:80 -p 443:443 susi


Deploying Susi on cloud with kubernetes

We will use Google Cloud Service platform for demonstration. Create your GCS account. Goto dashboard and click on compute engine.

Enable Billing and Create New Project named XYZ. Open the terminal, by clicking the Google cloud shell button on the top.

Please set the compute zone to your nearest zone by running the below command.

gcloud config set compute/zone us-central1-a


Now we need to create cluster, on which we deploy susi app. We do it by this command.

gcloud container clusters create hello-cluster --num-nodes=3


We need to get docker from dockerhub and push it to our project repo.We do it by these commands.

sudo docker pull jyothiraditya/susi_server
gcloud docker -- push <image-id> gcr.io/<project-id>/<name>

We run the docker image on cluster by following commands.

kubectl run susi-server --image=gcr.io/<project-id>/<name> --port=80
kubectl get pods

We expose the container to external traffic, with help of load balancer by the following command.

kubectl expose deployment susi-server --type="LoadBalancer --port=80"


We get the external ip-address to access susi from browser. By entering

kubectl get service susi-server


You can now view the app by going to “EXTERNAL-IP:80”.

References : Docker build , Kubernetes deployment, Google cloud deployment 

 

Continue ReadingDeploying SUSI.AI with Docker

Control flow of SUSI AI on Android and database management using Realm

While developing a chat based android application, one of the most important things is keeping track of user’s messages. Since the user might want to access them in the absence of Internet connectivity (i.e remotely) as well, storing them locally is also important.

In SUSI we are using Realm to keep things organized in a systematic manner and constructing model (or adding appropriate attributes) for every new data type which the application needs. Right now we have three main models namely ChatMessage, WebLink and WebSearchModel. These three java classes define the structure of each possible message.  ChatMessage evaluates and classifies incoming response from server either to be an image or map or pie chart or web search url or other valid types of response. WebSearchModel and WebLink models are there to manage those results which contains link to various web searches.

Various result based lists are maintained for smooth flow of application. Messages sent in absence of Internet are stored in a list – nonDelivered. All the messages have an attribute isDelivered which is set to true if and only if they have been queried, otherwise the attribute is set to false which puts it in the nonDelivered list. Once the phone is connected back to the internet and the app is active in foreground, the messages are sent to server, queried and we get the response back in the app’s database where the attributes are assigned accordingly.

 

I will explain a functionality below that will give a more clear view about our coding practices and work flow.

When a user long taps a message, few options are listed(these actions are defined in recycleradapters->ChatFeedRecyclerAdapter.java) from which you may select one. In the code, this triggers the method onActionsItemClicked(). In this Overridden method, we handle what happens when a user clicks on one of the following options from item menu. In this post I’ll be covering only about the star/important message option.

case R.id.menu_item_important:
    nSelected = getSelectedItems().size();
    if (nSelected >0)
    {
        for (int i = nSelected - 1; i >= 0; i--) 
        {
            markImportant(getSelectedItems().get(i));
        }
        if(nSelected == 1) 
        {
            Toast.makeText(context,nSelected+" message 
            marked 
            important",Toast.LENGTH_SHORT).show();
        } 
        else 
        {
            Toast.makeText(context, nSelected + " 
            messages marked important",                                                      
            Toast.LENGTH_SHORT).show();
        }
        Important = realm.where(ChatMessage.class).
        equalTo("isImportant",true)
        .findAll().sort("id");
        for(int i=0;i<important.size();++i)
            Log.i("message ","" + 
            important.get(i).getContent());
        Log.i("total ",""+important.size());
        actionMode.finish();
    }
return true;

We have the count of messages which were selected. Each message having a unique id is looped through and the attribute “isImportant” of each message object is modified accordingly. To modify this field, We call the method markImportant() and pass the id of message which has to be updated.

public void markImportant(final int position) {
    realm.executeTransaction(new Realm.Transaction() {
        @Override
        public void execute(Realm realm) {
            ChatMessage chatMessage = getItem(position);
            chatMessage.setIsImportant(true);
            realm.copyToRealmOrUpdate(chatMessage);
        }
    });
}

This method copies the instance of the message whose id it has received and updates the attribute “isImportant” and finally updates the message instance in the database.

Below given is the code for ImportantMessage activity which will help you understand properly how lists are used to query the database.

public class ImportantMessages extends AppCompatActivity {
 
    private Realm realm;
    private RecyclerView rvChatImportant;
    @Override
    protected void onCreate(Bundle savedInstanceState) {
       super.onCreate(savedInstanceState);
       realm = Realm.getDefaultInstance();
       rvChatImportant = (RecyclerView) findViewById 
                          (R.id.rv_chat_important);
       actionBar.setDisplayHomeAsUpEnabled(true);
       setChatBackground();
       setupAdapter();
 
        //call to other methods
    }
 
    private void setupAdapter() {
        rvChatImportant = (RecyclerView) findViewById 
                          (R.id.rv_chat_important);
        LinearLayoutManager linearLayoutManager = new 
                            LinearLayoutManager(this);
        linearLayoutManager.setStackFromEnd(true);
        rvChatImportant. 
        setLayoutManager(linearLayoutManager);
        rvChatImportant.setHasFixedSize(false);
        RealmResults<ChatMessage> importantMessages = 
        realm.where(ChatMessage.class). 
        equalTo("isImportant",true).findAll().sort("id");
        TextView tv_msg = (TextView) findViewById 
                          (R.id.tv_empty_list);
 
        if(importantMessages.size()!=0)
            tv_msg.setVisibility(View.INVISIBLE);
        else
            tv_msg.setVisibility(View.VISIBLE);
 
        ChatFeedRecyclerAdapter recyclerAdapter = new 
             ChatFeedRecyclerAdapter(Glide.with(this), this, 
             importantMessages, true);
        rvChatImportant.setAdapter(recyclerAdapter);
        rvChatImportant.addOnLayoutChangeListener(new 
        View.OnLayoutChangeListener() {
            @Override
            public void onLayoutChange(View view, int left, 
            int top, int right, int bottom,
            int oldLeft, int oldTop, int oldRight, int 
            oldBottom) {
                if (bottom < oldBottom) {
                    rvChatImportant.postDelayed(new 
                    Runnable() {
                        @Override
                        public void run() {
                            int scrollTo = 
                            rvChatImportant.getA 
                            dapter().getItemCount() - 1;
                            scrollTo = scrollTo >= 0 ? 
                            scrollTo : 0;                             
                            rvChatImportant. 
                            scrollToPosition(scrollTo);
                        }
                    }, 10);
                }
            }
        });
    }
}
Continue ReadingControl flow of SUSI AI on Android and database management using Realm

Deploying SUSI-Server on Digital Ocean

Create and Setup a Droplet

To create your first Droplet first login to your account. If you are using Github student developer pack then you will get $50 as Digital Ocean Credits.

The create button will be right there on the first page, click on “Create Droplet”
Then a page will open asking you some configurations of a server.

You are free to choose any Distribution and choose a size. I chose Ubuntu and $20/mo plan.
Now add information about the Data Center you want to use.

Add your SSH Keys. These keys will be used to authenticate you on remote login to your server.

Click Create. In a few seconds your droplet should be up.

Navigate to Droplets page and see that your droplet is live there. You should see your droplet name and IP there.

SSH to your server and Deploy SUSI

In order to connect to a remote Linux server via SSH, you must have following:

  • User name: The remote user to log in as. The default admin user, or Superuser, on most Linux servers is root
  • Password and/or SSH Key: The password that is used to authenticate the user that you are logging in as. If you added a public SSH key to your droplet when you created it, you must have the private SSH key of the key pair (and passphrase, if it has one)
  • Server IP address: This is the address that uniquely identifies your server on the Internet.

Open Terminal and run

ssh root@SERVER_IP_ADDRESS

You should be able to login to your server.

Now to run SUSI Server

(Step 1 to 10 to setup java and gradle, skip if you have done it before)

Visit SERVER_IP_ADDRESS:4000 and see that SUSI Server is running there.

To stop the server use:

bin/stop.sh

Continue ReadingDeploying SUSI-Server on Digital Ocean

Detecting and Fixing Memory leaks in Susi Android App

In the fast development of the Susi App, somehow developers missed out some memory leaks in the app. It is a very common mistake that developers do. Most new android developers don’t know much about Memory leaks and how to fix them. Memory leaks makes the app slower and causes crashes due to OutOfMemoryException. To make the susi app more efficient, it is advised to look out for these memory leaks and fix them. This post will focus on teaching developers who’ll be contributing in Susi android App or any other android app about the memory leaks, detecting them and fixing them.

What is a memory leak?

Android system manages memory allocation to run the apps efficiently. When memory runs short, it triggers Garbage Collector (GC) which cleans up the objects which are no longer useful, clearing up memory for other useful objects. But, suppose a case when a non-useful object is referenced from a useful object. In that case Garbage Collector would mark the non-useful object as useful and thus won’t be able to remove it, causing a Memory Leak.

Now, when a memory leak occurs, the app demands for memory from the android system but the android system can only give a certain amount of memory and after that point it will refuse to give more memory and thus causing a OutOfMemoryException and crashing the app. Even if sometime due to memory leaks, the app doesn’t crash but it surely will slow down and skip frames.

Now, few other questions arises, like “How to detect these leaks?” , “What causes these leaks ?” and “How can we fix these?” Let’s cover these one by one.

Detecting Memory Leaks

You can detect memory leaks in android app in two ways :

  1. Using Android Studio
  2. Using Leak Canary

In this post I’ll be describing the way to use Leak Canary to detect Memory Leaks. If you want to know about the way to use android studio to detect leaks, check out this link .

Using Leak Canary for Memory Leak detection :

Leak Canary is a very handy tool when it comes to detecting memory leaks. It shows the memory leak in an another app in the mobile itself. Just add these lines under the dependencies in build.gradle file.

debugCompile ‘com.squareup.leakcanary:leakcanary-android:1.5.1’
releaseCompile ‘com.squareup.leakcanary:leakcanary-android-no-op:1.5.1’
testCompile ‘com.squareup.leakcanary:leakcanary-android-no-op:1.5.1’

And this code here in the MainApplication.java file.

if (LeakCanary.isInAnalyzerProcess(this)) {
// This process is dedicated to LeakCanary for heap analysis.

// You should not init your app in this process.

return
;
}
LeakCanary.install(this);
// Normal app init code…

You are good to go. Now just run the app and if there is a memory leak you will get something like this. It dumps the memory in .hprof file and displays it in another app.

      

Causes of Memory Leaks

There are many causes of memory leaks. I will list a few top of my head but there can be more.

  1. Static Activities and Views : Defining a static variable inside the class definition of the Activity and then setting it to the running instance of that Activity. If this reference is not cleared before the Activity’s lifecycle completes, the Activity will be leaked.
  2. Listeners : When you register a listener, it is advised to unregister it in onDestroy() method to prevent memory leaks. It is not that prominent but may cause memory leaks.
  3. Inner Classes : If you create an instance of Inner Class and maintain a static reference to it, there is a chance of memory leak.
  4. Anonymous Classes : A leak can occur if you declare and instantiate an AsyncTask anonymously inside your Activity. If it continues to perform background work after the Activity has been destroyed, the reference to the Activity will persist and it won’t be garbage collected until after the background task completes.
  5. Handlers and Threads : The very same principle applies to background tasks declared anonymously by a Runnable object and queued up for execution by a Handler object.

Preventing and Fixing Memory Leaks

So, now you know what are the causes of these memory leaks. You just have to be a little more careful while implementing these. Here are some more tips to prevent or fix memory leaks :

  1. Be extra careful when dealing with Inner classes and Anonymous classes. Make them static wherever possible. Use a static inner class with a WeakReference to the outer class if that helps.
  2. Be very careful with a static variable in your activity class because it can reference your activity and cause leak. Be sure to remove the reference in onDestroy().
  3. Unregister all listeners in onDestroy() method.
  4. Always terminate worker threads you initiated on Activity onDestroy().
  5. Make sure that your allocated resources are all collected as expected. Do not always rely on Garbage Collector.
  6. Try using the context-application instead of a context-activity.

Conclusion

So, now if you want to contribute in Susi Android App and implement a feature in it, you can just check if there is a memory leak due to your implementation and fix it for better performance of the app. Also, if you find any other memory leak in the app, do report it on the issue tracker, fix it and make the Susi Android App more efficient.

Happy Coding!

Continue ReadingDetecting and Fixing Memory leaks in Susi Android App

How to teach SUSI skills calling an External API

SUSI is an intelligent  personal assistant. SUSI can learn skills to understand and respond to user queries better. A skill is taught using rules. Writing rules is an easy task and one doesn’t need any programming background too. Anyone can start contributing. Check out these tutorials and do watch this video to get started and start teaching susi.

SUSI can be taught to call external API’s to answer user queries.

While writing skills we first mention string patterns to match the user’s query and then tell SUSI what to do with the matched pattern. The pattern matching is similar to regular expressions and we can also retrieve the matched parameters using $<parameter number>$ notation.

Example :

My name is *
Hi $1$!

When the user inputs “My name is Uday” , it is matched with “My name is *” and “Uday” is stored in $1$. So the output given is “Hi Uday!”.

SUSI can call an external API to reply to user query. An API endpoint or url when called must return a JSON or JSONP response for SUSI to be able to parse the response and retrieve the answer.

Rule Format for a skill calling an external API

The rule format for calling an external API is :

<regular expression for pattern matching>
!console: <return answer using $object$ or $required_key$>
{
  “url”:<API endpoint or url>,
  “path”:$.<key in the API response to find the answer>,
}
eol
  • Url is the API endpoint to be called which returns a JSON or JSONP response.
    The parameters to the url if any can be added using $$ notation.
  • Path is used to help susi know where to look for the answer in the returned response.
    If the path points to a root element, then the answer is stored in $object$, otherwise we can query $key$ to get the answer which is a value to the key under the path.
  • eol or end of line indicates the end of the rule.

Understanding the Path Attribute

Let us understand the Path attribute better through some test cases.

In each of the test cases we discuss what the path should be and how to retrieve the answer for a given required answer from the json response of an API.

  1. API response in json :

    {
      “Key1” : “Value1”
    }
    

Required answer : Value1
Path : “$.Key1    =>   Retrieve Answer:  $object$

 

  1. API response in json :
{
  “Key1” : [{“Key11” : “Value11”}]
}

Required answer : Value11
Path : $.Key1[0]   =>  Retrieve Answer: $Key11$
Path : $.Key1[0].Key11   => Retrieve Answer: $object$

 

  1. API response in json :
{
  “Key1” : {“Key11” : “Value11”}
}


Required answer : Value11
Path : $.Key1  => Retrieve Answer:  $Key11$
Path : $.Key1.Key11  => Retrieve Answer: $object$

 

  1. API response in json :
{
  “Key1” : {
           “Key11” : “Value11”,
           “Key12” : “Value12”
         }
}

Required answer : Value11 , Value12
Path : $.Key1  => Retrieve Answer:  $Key11$ , $Key12$

Where to write these rules?

Now, since we know how to write rules let’s see where to write them.

We use etherpads to write and test rules and once we finish testing our rule we can push those rules to the repo.

Steps to open, write and test rules:

  1. Open a new etherpad with a desired name <etherpad name> at http://dream.susi.ai/
  2. Write your skills code in the etherpad following the code format explained above.
  3. Now, to test your skill let’s chat with susi. Start a conversation with susi at http://susi.ai/chat to test your skills.
  4. Load your skills by typing dream <etherpad name> and wait for a response saying dreaming enabled for <etherpad name>
  5. Test your skill and follow step 4 every time you make changes to the code in your etherpad.
  6. After you are done testing, type stop dreaming and if you are satisfied with your skill do send a PR to help susi learn.

Examples

Let us try an example to understand this better.

1. Plot of a TV Show

Tvmaze is an open  TV API that provides information about tv shows. Let us write a rule to know the plot of a tv show. We can find many such APIs. Check out this link listing few of them.

  1.  Open an etherpad at http://dream.susi.ai/ named tvshowplot.
  2.   Enter the code to query plot of a TV show in the etherpad at                           http://dream.susi.ai/p/tvshowplot
* plot of *|* summary of *
!console:$object$
{
  "url":"http://api.tvmaze.com/singlesearch/shows?q=$2$",
  "path":"$.summary"
}
eol
  1. Now lets test our skill by starting a conversation with susi at http://susi.ai/chat.
  • User Query: dream tvshowplot
    Response:  dreaming enabled  for tvshowplot
  • User Query: what is the plot of legion
    Response: Legion introduces the story of David Haller: Since he was a teenager, David has struggled with mental illness. Diagnosed as schizophrenic, David has been in and out of psychiatric hospitals for years. But after a strange encounter with a fellow patient, he’s confronted with the possibility that the voices he hears and the visions he sees might be real. He’s based on the Marvel comics character Legion, the son of X-Men founder Charles Xavier (played by Patrick Stewart and James McAvoy in the films), first introduced in 1985.

Intermediate Processing:

Pattern Matching : $1$ = “what is the” ; $2$ = “legion”

Url : http://api.tvmaze.com/singlesearch/shows?q=legion

API response:

{
  "id": 6393,
  "url": "http:\/\/www.tvmaze.com\/shows\/6393\/legion",
  "name": "Legion",
  "type": "Scripted",
  "language": "English",
  "genres": [
             "Drama",
             "Action",
             "Science-Fiction"
            ],
  "summary": "<p><strong>Legion<\/strong> introduces the story of David Haller: Since he was a teenager, David has struggled with mental illness. Diagnosed as schizophrenic, David has been in and out of psychiatric hospitals for years. But after a strange encounter with a fellow patient, he's confronted with the possibility that the voices he hears and the visions he sees might be real. He's based on the Marvel comics character Legion, the son of X-Men founder Charles Xavier (played by Patrick Stewart and James McAvoy in the films), first introduced in 1985.<\/p>",
  "updated": 1491955072,
}

Note: The API response has been trimmed to show only the relevant content.

Path : $.summary

Retrieving Answer: so our required answer in the api response is under the key summary and is retrieved using $object$ since it is a root element.

 

Screenshots:

2. Cooking Recipes

Let us try it out with another API.
Recipepuppy is an cooking recipe API where users can query various recipes.

  1.  Open a etherpad at http://dream.susi.ai/ named recipe.
  2.   Enter the code to query a recipe in the etherpad at  http://dream.susi.ai/p/recipe
#Gives recipes and links to cook a dish
* cook *
!console:<p>To cook  <strong>$title$</strong> : <br>The ingridients required are: $ingredients$. <br> For instruction to prepare the dish $href$ </p>
{
  "url":"http://www.recipepuppy.com/api/?q=$2$",
  "path":"$.results"
}
eol
  1. Now lets test our skill by starting a conversation with susi at http://susi.ai/chat.
  • User Query: dream recipe
    Response:  dreaming enabled  for recipe
  • User Query: how to cook chicken biryani
    Response: To cook Chicken Biryani Recipe :
    The ingridients required are: chicken, seeds, chicken broth, rice, butter, peas, garlic, red onions, cardamom, curry paste, olive oil, tomato, coriander, cumin, brown sugar, tumeric.
    For instruction to prepare the dish Click Here!

Intermediate Processing:

Pattern Matching : $1$ = “how to” ; $2$ = “chicken biryani”

Url : http://www.recipepuppy.com/api/?q=chicken biryani

API response:

{
  "title": "Recipe Puppy",
  "version": 0.1,
  "href": "http:\/\/www.recipepuppy.com\/",
  "results": [
              {
                "title": "Chicken Biryani Recipe",
                "href": "http:\/\/www.grouprecipes.com\/53040\/chicken-biryani.html",
                "ingredients": "chicken, seeds, chicken broth, rice, butter, peas, garlic, red onions, cardamom, curry paste, olive oil, tomato, coriander, cumin, brown sugar, tumeric",
                "thumbnail": "http:\/\/img.recipepuppy.com\/413822.jpg"
             },
           ]
}

Note: The API response has been trimmed to show only the relevant content.

Path : $.results[0]

Retrieving Answer: so our required answer in the api response is under the key results and since it’s an array we are using the first element of the array and since the element is a dictionary too we use its keys correspondingly to answer. The $href$ is rendered as “Click Here” hyperlinked to the actual url.

 

Screenshots:

 

We have successfully taught susi a skill which tells users about the plot of a tv show and a skill to query recipes.
Cheers!
Following similar procedure, we can make use of other APIs and teach susi several new skills.

Resources
Continue ReadingHow to teach SUSI skills calling an External API