Giving a Voice to Susi

Susi AI already has various apps and is available as a chatbot in various messaging platforms. We are going a step forward to make an SDK available for Susi that can be integrated on any Hardware Device (say speakers, toys, your bicycle etc - possibilities are endless ) One of the problem that I encountered while making a Prototype for the same is selecting an appropriate Text to Speech (TTS)  Engine. It was a challenge, since on platforms like Android and iOS , you may utilize TTS engines bundled with Platform easily via a platform specific API, which are well optimized and give good performance.  The same was difficult on a hardware device that can run only Linux with no TTS provided by default. Thus, I explored some possibilities eSpeak TTS: eSpeak TTS (http://espeak.sourceforge.net/) was the first option considered for the task. eSpeak is a compact open source software speech synthesizer for English and other languages, for Linux and Windows. The major advantage of eSpeak is its small size (2MB) and small memory footprint which is advantageous in Low Memory Hardware like Orange Pi Zero or Raspberry Pi Zero. Setting up eSpeak was easy but with its advantages , there were some drawbacks too. The voice synthesis was quite robotic. Very few voices were available. Festival TTS: Festival offers a general framework for building speech synthesis systems as well as including examples of various modules. As a whole it offers full text to speech through a number APIs: from shell level, though a Scheme command interpreter, as a C++ library, from Java, and an Emacs interface. Festival is free software. Festival and the speech tools are distributed under an X11-type licence allowing unrestricted commercial and non-commercial use alike. Installing Festival: On Arch Linux , it was pretty straight forward. sudo pacman -S festival There is a full wiki dedicated to it. ( https://wiki.archlinux.org/index.php/Festival ) Testing Festival Festival has an interpreter to test it out. It can be invoked using You may test out a TTS output using: festival> (SayText "Hi!! I am Susi") But the default sound in festival is still robotic and male. You don’t want your Personal Assistant to scare you out when you speak to her. Thus, I searched on what are the best female voices available for Festival. After looking at a discussion on the thread, https://ubuntuforums.org/showthread.php?t=751169 , I found that CMU-Arctic and HTS are some of the best voice sets for  Festival. In Arch Linux, additional voice packs, are supplied in two additional packages, festival-us and festival-english Installation is straightforward: sudo pacman -S festival-us festival-english Now, on festival REPL , we can test out our new voices. To see all available voices festival> (voice.list) (rab_diphone kal_diphone cmu_us_rms_cg cmu_us_awb_cg cmu_us_slt_cg) Testing out a voice festival> (voice_cmu_us_awb_cg) cmu_us_awb_cg festival> (SayText "Hi!! I am Susi") This way after testing out all voices, with many different phrases. cmu_us_slt_cg  felt like an appropriate voice. Setting Voice as Default Voice may be set as default by adding following line to .festivalrc…

Continue ReadingGiving a Voice to Susi

Implementing a chatbot using the SUSI.AI API

SUSI AI is an intelligent Open Source personal assistant. It is a server application which is able to interact with humans as a personal assistant. The first step in implementing a bot using SUSI AI is to specify the pathway for query response from SUSI AI server. The steps mentioned below provide a step-by-step guide to establish communication with SUSI AI server: Given below is HTML code that demonstrates how to connect with SUSI API through an AJAX call. To put this file on a Node Js server, see Step 2.  To view the response of this call, follow Step 4. <!DOCTYPE html> <body> <h1>My Header</h1> <p>My paragraph.</p> //Script with source here //Script to be written here </body> </html> In above code add scripts given below and end each script with closing tag </script>. In the second script we are calling SUSI API with hello query and showing data that we are receiving through call on console. <script src="https://ajax.googleapis.com/ajax/libs/jquery/3.2.1/jquery.min.js"> <script> $(function (){ $.ajax({ dataType: 'jsonp', type:'GET', url: 'http://api.susi.ai/susi/chat.json? timezoneOffset=-300&q=hello', success: function(data){ console.log('success', data); } }); }); Code below is in node js to setup localhost and getting the same above result on browser. Below is Node Js code to setup a server at localhost for the above created HTML file. var http = require('http'); var fs = require('fs'); http.createServer(function (req, res) {  fs.readFile('YOURFILENAME.html', function(err, data) {    res.writeHead(200, {'Content-Type': 'text/html'});    res.write(data);    res.end();  }); }).listen(9000); We will get following response by running this Node js code and checking results on http://localhost:9000/ To run this code install Node Js and write “node filename.js” in command line. You can open above window by right clicking on page and selecting Inspect. Go to the Network option and select the relevant api call from left section of Inspect window. We have successfully got response from SUSI API and now we can use this response for building bots for receiving replies for user.

Continue ReadingImplementing a chatbot using the SUSI.AI API

Hotword Detection in Susi Android

Hotword detection is one of the coolest features in Android. Voice control has emerged as a popular method for interacting with smartphones and wearable devices. It allows the user to interact with the app without even touching the device in a much intuitive way. At the same time it's difficult to implement them in the Android app and requires use of large number of resources. Let us dive deeper into this topic. So, Firstly What is a Hotword? Hotword is generally a phrase or a word that can be used to initiate a particular task in the app. The user needs to speak the phrase or the hotword which on detection makes a callback that can be used to carry out a particular task in the app. Examples of hotword include “Ok Google” which is used to initiate Google assistant in the Android mobile phones. Why is it difficult to implement hotword detection? To enable hotword detection is a cumbersome task. There are different problems associated with it including:- Change in the accent of the speaker. Continuous task of recognizing the voice and then matching it with the hotword. The features needs to be run as a service in android which causes the drain of battery. Also it is a memory intensive task at the same time. Let's us look at the present techniques available for the hotword detection. The Android provides a class called as AlwaysOnHotwordDetector. This class can be used to detect the keywords inside the activity of an Android app. For more details you can visit this link. The implementation goes like this /** * @param text The keyphrase text to get the detector for. * @param locale The java locale for the detector. * @param callback A non-null Callback for receiving the recognition events. * @param voiceInteractionService The current voice interaction service. * @param modelManagementService A service that allows management of sound models. * * @hide */ public AlwaysOnHotwordDetector(String text, Locale locale, Callback callback, KeyphraseEnrollmentInfo keyphraseEnrollmentInfo, IVoiceInteractionService voiceInteractionService, IVoiceInteractionManagerService modelManagementService) {   mText = text;   mLocale = locale;   mKeyphraseEnrollmentInfo = keyphraseEnrollmentInfo;   mKeyphraseMetadata = mKeyphraseEnrollmentInfo.getKeyphraseMetadata(text, locale);   mExternalCallback = callback;   mHandler = new MyHandler();   mInternalCallback = new SoundTriggerListener(mHandler);   mVoiceInteractionService = voiceInteractionService;   mModelManagementService = modelManagementService;   new RefreshAvailabiltyTask().execute(); } 2.  We can also detect hotword using Pocketsphinx library in Android. The library acts as a service in the Android app. It is quite efficient when it comes to battery consumption. The implementation goes like this:-     recognizer = defaultSetup()    .setAcousticModel(new File(assetsDir, "en-us-ptm"))    .setDictionary(new File(assetsDir, "cmudict-en-us.dict"))    .getRecognizer(); recognizer.addListener(this); // Create keyword-activation search. recognizer.addKeyphraseSearch(KWS_SEARCH, KEYPHRASE); // Create grammar-based searches. File menuGrammar = new File(assetsDir, "menu.gram"); recognizer.addGrammarSearch(MENU_SEARCH, menuGrammar); // Next search for digits File digitsGrammar = new File(assetsDir, "digits.gram"); recognizer.addGrammarSearch(DIGITS_SEARCH, digitsGrammar); // Create language model search. File languageModel = new File(assetsDir, "weather.dmp"); recognizer.addNgramSearch(FORECAST_SEARCH, languageModel); recognizer.startListening(searchName); Now Lets us look at how we are implementing it in Susi Android: In Susi Android, we have used CMU's Pocketsphinx library for the hotword detection. We are using “Hi Susi” as the hotword for the detection. private SpeechRecognizer recognizer;…

Continue ReadingHotword Detection in Susi Android

Deploying SUSI.AI with Docker

Docker is much more efficient than VM in allocating shared resources between various containers as shown in figure. To deploy SUSI we need to create docker container. There are two ways to build it. First way is fork the SUSI project in github. Then you can signup in dockerhub and create autobuild docker container. The second way is to manually build docker file from command prompt of your computer. The following instructions needs to be executed in cloud shell or linux machine. sudo apt-get update sudo apt-get upgrade sudo apt-get -y install docker.io sudo fallocate -l 4G /swapfile sudo chmod 600 /swapfile sudo mkswap /swapfile sudo swapon /swapfile sudo docker build https://github.com/fossasia/susi_server.git The first three commands install docker software to the machine. Next three lines give required permissions and execution abilities to docker. Final command builds docker container from github. Thus, We have successfully made docker container. We can deploy it on the cloud by using following command. sudo docker run -d -p 80:80 -p 443:443 susi Deploying Susi on cloud with kubernetes We will use Google Cloud Service platform for demonstration. Create your GCS account. Goto dashboard and click on compute engine. Enable Billing and Create New Project named XYZ. Open the terminal, by clicking the Google cloud shell button on the top. Please set the compute zone to your nearest zone by running the below command. gcloud config set compute/zone us-central1-a Now we need to create cluster, on which we deploy susi app. We do it by this command. gcloud container clusters create hello-cluster --num-nodes=3 We need to get docker from dockerhub and push it to our project repo.We do it by these commands. sudo docker pull jyothiraditya/susi_server gcloud docker -- push <image-id> gcr.io/<project-id>/<name> We run the docker image on cluster by following commands. kubectl run susi-server --image=gcr.io/<project-id>/<name> --port=80 kubectl get pods We expose the container to external traffic, with help of load balancer by the following command. kubectl expose deployment susi-server --type="LoadBalancer --port=80" We get the external ip-address to access susi from browser. By entering kubectl get service susi-server You can now view the app by going to “EXTERNAL-IP:80”. References : Docker build , Kubernetes deployment, Google cloud deployment   

Continue ReadingDeploying SUSI.AI with Docker

Control flow of SUSI AI on Android and database management using Realm

While developing a chat based android application, one of the most important things is keeping track of user’s messages. Since the user might want to access them in the absence of Internet connectivity (i.e remotely) as well, storing them locally is also important. In SUSI we are using Realm to keep things organized in a systematic manner and constructing model (or adding appropriate attributes) for every new data type which the application needs. Right now we have three main models namely ChatMessage, WebLink and WebSearchModel. These three java classes define the structure of each possible message.  ChatMessage evaluates and classifies incoming response from server either to be an image or map or pie chart or web search url or other valid types of response. WebSearchModel and WebLink models are there to manage those results which contains link to various web searches. Various result based lists are maintained for smooth flow of application. Messages sent in absence of Internet are stored in a list - nonDelivered. All the messages have an attribute isDelivered which is set to true if and only if they have been queried, otherwise the attribute is set to false which puts it in the nonDelivered list. Once the phone is connected back to the internet and the app is active in foreground, the messages are sent to server, queried and we get the response back in the app’s database where the attributes are assigned accordingly.   I will explain a functionality below that will give a more clear view about our coding practices and work flow. When a user long taps a message, few options are listed(these actions are defined in recycleradapters->ChatFeedRecyclerAdapter.java) from which you may select one. In the code, this triggers the method onActionsItemClicked(). In this Overridden method, we handle what happens when a user clicks on one of the following options from item menu. In this post I’ll be covering only about the star/important message option. case R.id.menu_item_important: nSelected = getSelectedItems().size(); if (nSelected >0) { for (int i = nSelected - 1; i >= 0; i--) { markImportant(getSelectedItems().get(i)); } if(nSelected == 1) { Toast.makeText(context,nSelected+" message marked important",Toast.LENGTH_SHORT).show(); } else { Toast.makeText(context, nSelected + " messages marked important", Toast.LENGTH_SHORT).show(); } Important = realm.where(ChatMessage.class). equalTo("isImportant",true) .findAll().sort("id"); for(int i=0;i<important.size();++i) Log.i("message ","" + important.get(i).getContent()); Log.i("total ",""+important.size()); actionMode.finish(); } return true; We have the count of messages which were selected. Each message having a unique id is looped through and the attribute “isImportant” of each message object is modified accordingly. To modify this field, We call the method markImportant() and pass the id of message which has to be updated. public void markImportant(final int position) { realm.executeTransaction(new Realm.Transaction() { @Override public void execute(Realm realm) { ChatMessage chatMessage = getItem(position); chatMessage.setIsImportant(true); realm.copyToRealmOrUpdate(chatMessage); } }); } This method copies the instance of the message whose id it has received and updates the attribute “isImportant” and finally updates the message instance in the database. Below given is the code for ImportantMessage activity which will help you understand properly how lists are used…

Continue ReadingControl flow of SUSI AI on Android and database management using Realm

Deploying SUSI-Server on Digital Ocean

Create and Setup a Droplet To create your first Droplet first login to your account. If you are using Github student developer pack then you will get $50 as Digital Ocean Credits. The create button will be right there on the first page, click on "Create Droplet" Then a page will open asking you some configurations of a server. You are free to choose any Distribution and choose a size. I chose Ubuntu and $20/mo plan. Now add information about the Data Center you want to use. Add your SSH Keys. These keys will be used to authenticate you on remote login to your server. Click Create. In a few seconds your droplet should be up. Navigate to Droplets page and see that your droplet is live there. You should see your droplet name and IP there. SSH to your server and Deploy SUSI In order to connect to a remote Linux server via SSH, you must have following: User name: The remote user to log in as. The default admin user, or Superuser, on most Linux servers is root Password and/or SSH Key: The password that is used to authenticate the user that you are logging in as. If you added a public SSH key to your droplet when you created it, you must have the private SSH key of the key pair (and passphrase, if it has one) Server IP address: This is the address that uniquely identifies your server on the Internet. Open Terminal and run ssh root@SERVER_IP_ADDRESS You should be able to login to your server. Now to run SUSI Server (Step 1 to 10 to setup java and gradle, skip if you have done it before) git clone https://github.com/fossasia/susi_server.git cd susi_server sudo apt-get update sudo apt-get install default-jdk sudo apt-get install default-jre wget https://services.gradle.org/distributions/gradle-3.4.1-bin.zip sudo mkdir /opt/gradle sudo unzip -d /opt/gradle gradle-3.4.1-bin.zip export PATH=$PATH:/opt/gradle/gradle-3.4.1/bin gradle build bin/start.sh Visit SERVER_IP_ADDRESS:4000 and see that SUSI Server is running there. To stop the server use: bin/stop.sh

Continue ReadingDeploying SUSI-Server on Digital Ocean

Detecting and Fixing Memory leaks in Susi Android App

In the fast development of the Susi App, somehow developers missed out some memory leaks in the app. It is a very common mistake that developers do. Most new android developers don’t know much about Memory leaks and how to fix them. Memory leaks makes the app slower and causes crashes due to OutOfMemoryException. To make the susi app more efficient, it is advised to look out for these memory leaks and fix them. This post will focus on teaching developers who’ll be contributing in Susi android App or any other android app about the memory leaks, detecting them and fixing them. What is a memory leak? Android system manages memory allocation to run the apps efficiently. When memory runs short, it triggers Garbage Collector (GC) which cleans up the objects which are no longer useful, clearing up memory for other useful objects. But, suppose a case when a non-useful object is referenced from a useful object. In that case Garbage Collector would mark the non-useful object as useful and thus won’t be able to remove it, causing a Memory Leak. Now, when a memory leak occurs, the app demands for memory from the android system but the android system can only give a certain amount of memory and after that point it will refuse to give more memory and thus causing a OutOfMemoryException and crashing the app. Even if sometime due to memory leaks, the app doesn’t crash but it surely will slow down and skip frames. Now, few other questions arises, like “How to detect these leaks?” , “What causes these leaks ?" and “How can we fix these?” Let’s cover these one by one. Detecting Memory Leaks You can detect memory leaks in android app in two ways : Using Android Studio Using Leak Canary In this post I’ll be describing the way to use Leak Canary to detect Memory Leaks. If you want to know about the way to use android studio to detect leaks, check out this link . Using Leak Canary for Memory Leak detection : Leak Canary is a very handy tool when it comes to detecting memory leaks. It shows the memory leak in an another app in the mobile itself. Just add these lines under the dependencies in build.gradle file. debugCompile 'com.squareup.leakcanary:leakcanary-android:1.5.1' releaseCompile 'com.squareup.leakcanary:leakcanary-android-no-op:1.5.1' testCompile 'com.squareup.leakcanary:leakcanary-android-no-op:1.5.1' And this code here in the MainApplication.java file. if (LeakCanary.isInAnalyzerProcess(this)) { // This process is dedicated to LeakCanary for heap analysis. // You should not init your app in this process. return; } LeakCanary.install(this); // Normal app init code... You are good to go. Now just run the app and if there is a memory leak you will get something like this. It dumps the memory in .hprof file and displays it in another app.        Causes of Memory Leaks There are many causes of memory leaks. I will list a few top of my head but there can be more. Static Activities and Views : Defining a static variable inside the class definition of…

Continue ReadingDetecting and Fixing Memory leaks in Susi Android App

How to teach SUSI skills calling an External API

SUSI is an intelligent  personal assistant. SUSI can learn skills to understand and respond to user queries better. A skill is taught using rules. Writing rules is an easy task and one doesn’t need any programming background too. Anyone can start contributing. Check out these tutorials and do watch this video to get started and start teaching susi. SUSI can be taught to call external API’s to answer user queries. While writing skills we first mention string patterns to match the user's query and then tell SUSI what to do with the matched pattern. The pattern matching is similar to regular expressions and we can also retrieve the matched parameters using $<parameter number>$ notation. Example : My name is * Hi $1$! When the user inputs “My name is Uday” , it is matched with “My name is *” and “Uday” is stored in $1$. So the output given is “Hi Uday!”. SUSI can call an external API to reply to user query. An API endpoint or url when called must return a JSON or JSONP response for SUSI to be able to parse the response and retrieve the answer. Rule Format for a skill calling an external API The rule format for calling an external API is : <regular expression for pattern matching> !console: <return answer using $object$ or $required_key$> { “url”: “<API endpoint or url>”, “path”: “$.<key in the API response to find the answer>”, } eol Url is the API endpoint to be called which returns a JSON or JSONP response. The parameters to the url if any can be added using $$ notation. Path is used to help susi know where to look for the answer in the returned response. If the path points to a root element, then the answer is stored in $object$, otherwise we can query $key$ to get the answer which is a value to the key under the path. eol or end of line indicates the end of the rule. Understanding the Path Attribute Let us understand the Path attribute better through some test cases. In each of the test cases we discuss what the path should be and how to retrieve the answer for a given required answer from the json response of an API. API response in json : { “Key1” : “Value1” } Required answer : Value1 Path : “$.Key1    =>   Retrieve Answer:  $object$   API response in json : { “Key1” : [{“Key11” : “Value11”}] } Required answer : Value11 Path : $.Key1[0]   =>  Retrieve Answer: $Key11$ Path : $.Key1[0].Key11   => Retrieve Answer: $object$   API response in json : { “Key1” : {“Key11” : “Value11”} } Required answer : Value11 Path : $.Key1  => Retrieve Answer:  $Key11$ Path : $.Key1.Key11  => Retrieve Answer: $object$   API response in json : { “Key1” : { “Key11” : “Value11”, “Key12” : “Value12” } } Required answer : Value11 , Value12 Path : $.Key1  => Retrieve Answer:  $Key11$ , $Key12$ Where to write these rules? Now, since we know…

Continue ReadingHow to teach SUSI skills calling an External API

Deploying Susi Server on Google Cloud with Kubernetes

Susi (acronym for Scientific User Support Intelligence) is an advanced AI made by people at FOSSASIA. It is an AI made by the people and for the people. Susi is an Open Source Project under LGPL Licence. SUSI.AI already has many Skills and anyone can add new skills through simple console rules. If you want to participate in the development of the SUSI server you can start by learning to deploy it on a cloud system like Google Cloud. This way whenever you make a change to Susi Server, you can test it out on various Susi Apps instantly. Google Cloud with Kubernetes provide this ability. Let’s dig deep into what is Google Cloud Platform and Kubernetes. What is Google Cloud Platform ? Google Cloud Platform lets you build and host applications and websites, store data, and analyze data on Google’s scalable infrastructure. Google Cloud Platform (at the time of writing this article) also provides free credits worth $300 for 1 year for testing out the Platform and test your applications. What is Kubernetes ? Kubernetes is an open-source system for automatic deployment, management and scaling of containerized applications. It makes it easy to roll out updates to your application with simple commands from your development machine and scale horizontally easily by adding more clusters as demand increase. Deploying Susi Server on Kubernetes Deploying Susi Server on Kubernetes is a fairly easy task. Follow up the steps to get it running. Create a Google Cloud Account Sign up for a Google Cloud Account (https://cloud.google.com/free-trial/) and get 300$ credits for initial use. Create a New Project After successful sign up, create a new project on Google Cloud Console. Let’s name it Susi-Kubernetes .  You will be provided a ProjectID. Remember it for further reference. Install Google Cloud SDK and kubectl Go to https://cloud.google.com/sdk/ and see instructions to setup Google Cloud SDK on your respective OS. After Google Cloud SDK install, run gcloud components install kubectl This will install kubectl for interacting with Kubernetes. Login and setup project Login to your Google Cloud Account using $ gcloud auth login 2. List all the projects using $ gcloud config list project [core] project = <PROJECT_ID> 3. Select your project $ gcloud config set project <PROJECT_ID> 4. Install JDK8 for susi_server setup and set it as default. 5. Clone your fork of the Susi Server Repository $ git clone https://github.com/<your_username>/susi_server.git $ cd susi_server/ 6. Build project and run Susi Server locally $ ./gradlew build $ bin/start.sh Susi server must have been started started and web interface is accessible on http://localhost:4000 Install Docker and build Docker image for Susi Install Docker. Debian and derivatives:  sudo apt install docker Arch Linux:   sudo pacman -S docker  Build Docker Image for Susi $ docker build -t gcr.io/<Project_id>/susi:v1 . Push Image to Google Container Registry private to your project. $ gcloud docker -- push gcr.io/<Project_id>/susi:v1 Create Cluster and Deploy your Susi Server there Create Cluster. You may specify different zone, number of nodes and machine type depending upon requirement. $ gcloud container clusters create <Cluster-Name> --num-nodes…

Continue ReadingDeploying Susi Server on Google Cloud with Kubernetes

Susi AI Skill Development

What is Susi? Susi is an open source intelligent personal assistant which has the capability to learn and respond better to queries. It is also capable of making to-do lists, setting alarms, providing weather and traffic info all in real time. Susi responds based on skills. What is a skill? How do we teach a skill? A skill is a piece of code which performs a set of actions in order to respond to the user’s query. These skills are based on pattern matching which help them mapping the user’s query to a specific skill and responding accordingly. Teaching a skill to Susi is surprisingly very easy to implement. One can take a look at the Susi Skill Development Tutorial and a video workshop by Michael Christen. I will try to give a basic idea on how to create a skill, it’s basic structure and some of the skills I developed in the first week. Prepare to create a skill: Head over to http://dream.susi.ai Create a etherpad with some relevant name Delete all text currently present in there Start writing your skill Adding to this, for testing a skill one can head over to Susi Web Chat Interface. Basic Structure for calling an API: <Regular expression to be matched here> !console:<response given to the user> { "url":"<API endpoint>", "path":"<Json path here>" } eol So, let me explain this line by line. The regular expression is the one to which the user’s query is matched first. The console is meant to output the actual response the user sees as response. In place of the “url”, the API endpoint is passed in. “path” here specifies how we traverse through the response Json or Jsonp to get the object, starts with “$.”. At last, “eol” which is the end-of-line marks the end of a skill. Let’s take an example for better understanding of this: random gif !console: $url$ {    "url" : "http://api.giphy.com/v1/gifs/trending?api_key=dc6zaTOxFJmzC",    "path" : "$.data[0].images.fixed_height" } eol   This skill responds with a link to a random gif. Steps involved: Match the string “random gif” with the user’s query. On successful match, make an API call to the API endpoint specified in “url” On response, extract the object at the specified path in the json under “path” Respond to the user with the “url” key’s value which would here be an URL of a GIF. Let’s try it out on Susi Web Chat. For this, you will first have to load your skill using the dream command followed by etherpad name: dream <etherpad name>. And then you can start testing your skill. So, we queried “random gif” and we got a response “Click Here!”. The complete URL didn’t show up because all the URLs are currently parsed and a hyperlink for each is created. So try clicking on it to find a GIF.   Now, let’s look at one more skill I developed during this period. # Returns the name of the president of a country president of *|who is the president of…

Continue ReadingSusi AI Skill Development