Integration of SUSI AI to Alexa

An Alexa skill which can be used to ask susi for answers like: “Alexa, ask susi chat who are you” or “Alexa, ask susi chat what is the temperature in berlin”.

If at any point of time, you are unclear about the code in the blog post, you can check the code of the already made SUSI Alexa skill from the susi_alexa_skill repository.

Getting Started : Alexa Susi AI Skill

Follow the instructions below:

Visit the Amazon developer site and Login.

Click Alexa, on the top bar.

Click Alexa skills kit.

Click on add a new skill button on the top right of the page.

We will be at the skill information tab.

 

Write the name of the skill Write the invocation name of the skill i.e. the name that will be used to trigger your skill. Like in our case, if we need to ask anything (as we have ‘susi chat’ as the invocation name), we will ask with “Alexa, ask susi chat” as a prefix to our question sentence.

By clicking next, we will be redirected to the second tab i.e. Interaction model. We need to fill two fields here i.e. intent schema and sample utterances. For intent schema, we need to write all the available intents and the parameters for each of them. Like in our case:

{
 "intents": 
 [
   {
     "slots": [
       {
         "name": "query",
         "type": "AMAZON.LITERAL"
       }
     ],
     "intent": "callSusiApi"
   }
 ]
}

We have a single intent that is “callSusiApi” and the parameter it accepts is “query” of type “AMAZON.LITERAL” (in simple words, a string type). Parameters are termed as slots here. The different types of slots available, can be seen from here.

For sample utterances, we need to tell what utterances by the client will lead to what intent. In our case:

We have just one intent and the whole string uttered by the client should be fed to this intent as a “query” slot (parameter).

Let’s click next now.

We will be shifted to the configuration tab.

We will be making a lambda function, which will hold the code for our Susi skill, further we need to link that code to this skill. To do the linking we need to get the Amazon resource name i.e. ARN and fill it in the field named endpoint:


To get the amazon resource name, in a new tab, visit here. Visit “Lambda” followed by get started button. Click on “Create a lambda function”:

We need to select a blueprint for our lambda function. Select the “blank function” option for that.


Click next.

For configure triggers option, click this box and select “Alexa skills kit” option.


Click next.

In configure function tab, just write the name of the function and its description. Let’s code our lambda function:

// basic syntax that should be available in the lambda function
var https = require('http');
exports.handler = (event, context) => {
  try {
    if (event.session.new) {
      // New Session
      console.log("NEW SESSION")
    }
    switch (event.request.type) {
      case "LaunchRequest":
        // Launch Request
        console.log(`LAUNCH REQUEST`)
        context.succeed(
          generateResponse(
            buildSpeechletResponse("Welcome to a Susi Skill, this is an A.I. chatbot developed by Fossasia open source community. Ask anything to me like temperature at a place or rating of a movie or any other thing which you would like to ask?", false),
            {}
          )
        )
        break;
      case "IntentRequest":
        // Intent Request
        console.log(`INTENT REQUEST`)

        switch(event.request.intent.name) {
          case "callSusiApi":
            console.log(event.request.intent.slots.query.value)
            var endpoint = "http://api.susi.ai/susi/chat.json?q="+event.request.intent.slots.query.value; // ENDPOINT GOES HERE
            var body = ""
            https.get(endpoint, (response) => {
              response.on('data', (chunk) => { body += chunk })
              response.on('end', () => {
                var data = JSON.parse(body)
                // fetching answer from susi
                var viewCount = data.answers[0].actions[0].expression;
                if(viewCount.indexOf('I found this on the web') != -1)
                    viewCount = 'I have no idea about it, sorry.';
                context.succeed(
                  generateResponse(
                    buildSpeechletResponse(`${viewCount}`, false),
                    {}
                  )
                )
              })
            })
            break;

          default:
            throw "Invalid intent"
        }

        break;
      case "SessionEndedRequest":
        // Session Ended Request
        console.log(`SESSION ENDED REQUEST`)
        break;
      default:
        context.fail(`INVALID REQUEST TYPE: ${event.request.type}`)
    }
  } catch(error) { context.fail(`Exception: ${error}`) }
}

// Helpers
buildSpeechletResponse = (outputText, shouldEndSession) => {
  return {
    outputSpeech: {
      type: "PlainText",
      text: outputText
    },
    shouldEndSession: shouldEndSession
  }
}

generateResponse = (speechletResponse, sessionAttributes) => {
  return {
    version: "1.0",
    sessionAttributes: sessionAttributes,
    response: speechletResponse
  }
}

Paste this code into the space given below “lambda function code”. In lambda function handler and role, Click the field named role and select “create a custom role” from the dropdown shown.

You will be redirected to a new page. Select the IAM role as lambda_basic_execution:

Click allow button in the bottom right. We will be redirected back to our previous page. We don’t need to worry about other settings on this page.

Click next.

Again cross-check the details shown and click next.

Now we will have our ARN (Amazon resource name) on the top right of the page.


Copy that and paste it into the field “endpoint” on our previously open browser tab:


Click next.

Great that our SUSI AI skill is ready!

Now we can test it with a sample query, when we get redirected to the test tab:


Also we can test it using our voice through reverb app available on play store or echosim by logging into your amazon account.

Till now, the skill can just be invoked or tested from your own amazon id. To make this skill public , you need to fill the other 2 tabs left that are “publishing information” and “privacy and compliance”.

Some sample strings that we can speak to test it: “Alexa, ask susi chat where are you” “Alexa, ask susi chat tell me a joke” “Alexa, ask susi chat what is a table” (where ‘susi chat’ is the invocation name).

This was the video that helped a lot in making this skill for Alexa. It can be referred too.

Continue ReadingIntegration of SUSI AI to Alexa

Getting user Location in SUSI Android App and using it for various SUSI Skills

Using user location in skills is a very common phenomenon among various personal assistant like Google Assistant, Siri, Cortana etc. SUSI is no different. SUSI has various skills which uses user’s current location to implement skills. Though skills like “restaurant nearby” or “hotels nearby” are still under process but skills like “Where am I” works perfectly indicating SUSI has all basic requirements to create more advance skills in near future.

So let’s learn about how the SUSI Android App gets location of a user and sends it to SUSI Server where it is used to implement various location based skills.

Sources to find user location in an Android app

There are three sources from which android app gets users location :

  1. GPS
  2. Network
  3. Public IP Address

All three of these have various advantages and disadvantages. The SUSI Android app uses cleverly each of them to always get user location so that it can be used anytime and anywhere.

Some factors for comparison of these three sources :

Factors GPS Network IP Address
Source Satellites Wifi/Cell Tower Public IP address of user’s mobile
Accuracy Most Accurate (20ft) Moderately Accurate (200ft) Least Accurate (5000+ ft)
Requirements GPS in mobile Wifi or sim card Internet connection
Time taken to give location Takes long time to get location Fastest way to get location Fast enough (depends on internet speed)
Battery Consumption High Medium Low
Permission Required User permission required User permission required No permission required
Location Factor Works in outdoors. Does not work near tall buildings Works everywhere Works everywhere

Implementation of location finding feature in SUSI Android App

SUSI Android app very cleverly uses all the advantages of each location finding source to get most accurate location, consume less power and find location in any scenario.

The /susi/chat.json endpoint of SUSI API requires following 7 parameters :

Sno. Parameter Type Requirement
1 q String Compulsory
2 timezoneOffset int Optional
3 longitude double Optional
4 latitude double Optional
5 geosource String Optional
6 language Language  code Optional
7 access_token String Optional

In this blog we will be talking about latitude , longitude and geosource. So, we need these three things to pass as parameters for location related skills. Let’s see how we do that.

Finding location using IP Address: At the starting of app, user location is found by making an API call to ipinfo.io/json . This results in following JSON response having a field “loc” giving location of user (latitude and longitude.

{
  "ip": "YOUR_IP_ADDRESS",
  "city": "YOUR_CITY",
  "region": "YOUR_REGION",
  "country": "YOUR_COUNTRY_CODE",
  "loc": "YOUR_LATITUDE,YOUR_LONGITUDE",
  "org": "YOUR_ISP"
}

By this way we got latitude, longitude and geosource will be “ip” . We find location using IP address only once the app is started because there is no need of finding it again and again as making network calls takes time and drains battery.

So, now we have user’s location but this is not accurate. So, we will now proceed to check if we can find location using network is more accurate than location using IP address.

Finding location using Network Service Provider : To actually use the network provider and find out location requires ACCESS_COARSE_LOCATION permission from user which can be asked during the run time. Also, the location can only be found out using this if user has his location setting is enabled. So, we check following condition.

if (ActivityCompat.checkSelfPermission(mContext, Manifest.permission.ACCESS_COARSE_LOCATION) == PackageManager.PERMISSION_GRANTED) {
}

If permission is granted by user to find location using network provider, we use following code snippet to find location. It updates location of user after every 5 minutes or 10 meters (whichever is achieved first).

locationManager.requestLocationUpdates(LocationManager.NETWORK_PROVIDER, 5 * 60 * 1000, 10, this);
location = locationManager.getLastKnownLocation(LocationManager.NETWORK_PROVIDER);
if (location != null) {
   source = "network";
   canGetLocation = true;
   latitude = location.getLatitude();
   longitude = location.getLongitude();
}

So, whenever we are about to send query to SUSI Server, we take location from Network services, thus updating previous values of latitude, longitude and geosource (found using IP address) with the new values (found using Network Provider), provided the user has granted permission. So, we now have location is from Network Provider which is more accurate than location from IP address. Now we will check if we can find location from GPS or not.

Finding location using GPS Service Provider : Finding location from GPS Provider is almost same as Network Provider. To find location using GPS Provider user must give  ACCESS_FINE_LOCATION permission. We just check if GPS of user is enabled and user has given permission to use GPS and also if GPS can actually provide location. Sometimes, GPS can not provide location because user is indoor. In that cases we leave location from GPS.

So, now if we update previous values of latitude, longitude and geosource (found using Network Provider) with the new values (found using GPS Provider) and send query to SUSI Server.

Summary

To send location to server for location skills, we need latitude, longitude and geosource. We first find these 3 things using IP address (no that accurate). So, geosource will be “ip” for now. Then check if we can find values using network provider. If yes, we update those 3 values with the ones got from network Provider (more accurate). Geosource will change to “network”. Finally, we check if we can find values using GPS provider. If yes, we update those 3 values with the ones got from GPS Provider (most accurate). Geosource will change to “gps”. So, by this way we can find location of user in any circumstance possible. If you want to use location in your app too. Just follow the above steps and you are good to go.

Resources

 

Continue ReadingGetting user Location in SUSI Android App and using it for various SUSI Skills

Understanding the working of SUSI Hardware

Susi on Hardware is the latest addition to full suite of SUSI Apps. Being a hardware project, one might feel like it is too much complex, however it is not the case.

The solution is being primary built on a Raspberry Pi which, however small it may be, is a computer. Most things you expect to work on a normal computer, work on Raspberry Pi as well with a few advantages being its small size and General Purpose I/O access. But it comes with caveats of an ARM CPU, which may not support all applications which are mainly targeted for x86.
There are a few other development boards from Intel as well, which use x86/x64 architecture.

While working on the project, I did not wanted to make it too generic for a board or set of Hardware, thus all components used were targeted to be cross-platform.

Components that make Susi Hardware

SUSI Server

SUSI Server is the foremost important thing in any SUSI Project. SUSI Server handles all the queries by user which can be supplied using REST API and supplies answer in a nice format for clients. It also provides AAA: Authentication, Authorization and Accounting support for managing user accounts across platforms.

Github Repository: https://github.com/fossasia/susi_server

Susi Python Library

Susi Python Library was developed along with Susi Hardware project. It can work independent of Hardware Project and can be included in any Python Project for Susi Intelligence. It provides easy access to Susi Server REST API through easy python methods.
Github Repository: https://github.com/fossasia/susi_api_wrapper

Python Speech Recognition Library

The best advantage of using Python is that in most cases , you do not need to re-invent the wheel, some already has done the work for you. Python Speech Recognition library support for speech recognition through microphone and by a voice sample. It supports a number of Speech API providers like Google Speech API. Wit.AI, IBM Watson Speech-To-Text and a lot more.
This provides free to choose any of the speech recognition providers. For now, we are using Google Speech API and IBM Watson Speech API.


Pypi Package: https://pypi.python.org/pypi/SpeechRecognition/Github Repository: https://github.com/Uberi/speech_recognition

PocketSphinx for Hotword Detection

CMU PocketSphinx is an open-source offline speech recognition library. We have used PocketSphinx to enable hotword detection to Susi Hardware so you can interact with Susi handsfree.
More information on its working can be found in my other blog post.

Github Repository: https://github.com/cmusphinx/pocketsphinx

Flite Speech Synthesis System

CMU Flite (Festival-Lite) is a small sized , fast and open source speech synthesis engine developed by Carnegie Mellon University .
More information of integration and usage in Susi can be found in my other blog post

Project Website: http://www.festvox.org/flite/

The whole working of all these components together can be explained using the Diagram below.

Continue ReadingUnderstanding the working of SUSI Hardware

Hotword Detection for SUSI Android with CMUsphinx

Being an AI for conversational bots, Hotword detection of SUSI is the top priority to the community. Another requirement was that there should be an option for an offline hotword detection. So, I was searching for an API that has all these capabilities. Sphinx by CMU was the obvious choice. It provides robust mechanism for hotword detection.

What is CMUsphinx?

CMUsphinx is open source and leading speech recognition toolkit. CMUsphinx has different modules for different tasks it needs to perform. Our requirement for SUSI is, that is needs to be lightweight, So we are using Pocketsphinx. Before going into integration let us discuss about basics of speech recognition.

Let us dive into coding and integrating Susi with pocketsphinx.

Building Pocketsphinx .AAR file

Git clone the sphinxbase, pocketsphinx and pocketsphinx-android and put them in the same folder. By following commands below.

git clone http://github.com/cmupshinx/sphinxbase
git clone http://github.com/cmupshinx/pocketsphinx
git clone http://github.com/cmupshinx/pocketsphinx-android

Then import pocketsphinx Android into Android studio. Run the project. .aar files pocketsphinx-android-5prealpha-debug.aar & pocketsphinx-android-5prealpha-release.aar  will be created in the build/outputs/aar.

Integrating Susi with Pocketsphinx

In Android Studio you need to the above generated AAR into your project. Just go to File > New > New module and choose Import .JAR/.AAR Package. After this, We need to change permissions of project. Add the following permissions in AndroidManifest.xml.

<uses-permission android:name="android.permission.WRITE_EXTERNAL_STORAGE" />
<uses-permission android:name="android.permission.RECORD_AUDIO" />

Import the following functions into your main activity.

import edu.cmu.pocketsphinx.Assets;
import edu.cmu.pocketsphinx.Hypothesis;
import edu.cmu.pocketsphinx.RecognitionListener;
import edu.cmu.pocketsphinx.SpeechRecognizer;
import edu.cmu.pocketsphinx.SpeechRecognizerSetup;

Next we need to sync the assets we get from .aar file in to our project. Edit app/build.gradle build file to run assets.xml. We do it by adding following code to build.gradle.

ant.importBuild 'assets.xml'
preBuild.dependsOn(list, checksum)
clean.dependsOn(clean_assets)

Now all the import and sync errors of gradle must disappear and you should be good to go. You can start your recognizer by adding this code to your activity.

recognizer = defaultSetup()
        .setAcousticModel(new File(assetsDir, "en-us-ptm"))
        .setDictionary(new File(assetsDir, 
"cmudict-en-us.dict"))
        .getRecognizer();
recognizer.addListener(this);

Decoder model is lengthy process that contains many operations, so it’s recommended to run in inside async task. These are commands for decoder to run. These commands essentially do acoustic and language modelling of speech.

// Create keyword-activation search.
recognizer.addKeyphraseSearch(KWS_SEARCH, KEYPHRASE);

// Create grammar-based searches.
File menuGrammar = new File(assetsDir, "menu.gram");
recognizer.addGrammarSearch(MENU_SEARCH, menuGrammar);

// Next search for digits
File digitsGrammar = new File(assetsDir, "digits.gram");
recognizer.addGrammarSearch(DIGITS_SEARCH, digitsGrammar);

// Create language model search.
File languageModel = new File(assetsDir, "weather.dmp");
recognizer.addNgramSearch(FORECAST_SEARCH, languageModel);

Speech recognition will end at onEndOfSpeech callback of the recognizer listener.  We can call recognizer.stop or recognizer.cancel(). Cancel will cancel the recognition, stop will cause the final result be passed you in onResult callback. During the recognition, you will get partial results in onPartialResult callback.

Now we have integrated Pocketsphinx with SUSI.AI in Android.

Continue ReadingHotword Detection for SUSI Android with CMUsphinx

Map Responses from SUSI Server

Giving user responses in Maps is a very common reply of various assistants. Android and iOS clients of SUSI.AI can Render their own custom map views but there is a challenge in doing that in Bots and other clients which can only render Images.

The MapServlet in SUSI-Server is a servlet that takes the constraints of a map as input parameters and returns PNG Image of the rendered map.

This is a simple HttpServlet. Since it does not requires any auth or user rights we do not extend AbstactAPIHandler in this class.

 
public class MapServlet extends HttpServlet {

This is to method which runs whenever the Servlet receives a GET Query. After getting the query the function call another function “process(…)” which processes the input parameters.

 
@Override
protected void doGet(HttpServletRequest request, HttpServletResponse          response) throws ServletException, IOException {
Query post = RemoteAccess.evaluate(request);

This line is the for DDOS Protection.  It checks if the user query rate is too high and rate limit his queries. If Server suspects that the post is a DDOS then it returns a 503 error.

if (post.isDoS_blackout()) {
  response.sendError(503, "your request frequency is too high"); return;
} 
process(request, response, post);

The process method get the zoom, latitude, longitude, width and the height of the map and checks for the errors in them.

if (map.getHeight() > height || map.getWidth() > width) {
            BufferedImage bi = map.getImage();

            int xoff = (map.getWidth() - width) / 2;
            int yoff = (map.getHeight() - height) / 2;

            bi = bi.getSubimage(xoff, yoff, width, height);
            map = new RasterPlotter(width, height, RasterPlotter.DrawMode.MODE_REPLACE, "FFFFFF");
            map.insertBitmap(bi, 0, 0);
        }

Then we compute our image using RasterPlotter.

map.setDrawMode(DrawMode.MODE_SUB);
map.setColor(0xffffff);
if (text.length() > 0) PrintTool.print(map, 6, 12, 0, uppercase ? text.toUpperCase() : text, -1, false, 100);
PrintTool.print(map, map.getWidth() - 6, map.getHeight() - 6, 0, "MADE WITH LOKLAK.ORG", 1, false, 50);

Here we calculate the height of the map. If a user passed a height and width in get parameters, we scale the map according to that but if not its 256×256.

    int mx = (int) (map.getWidth() * (lon - west_lon) / (east_lon - west_lon));
    int my = (int) (map.getHeight() * (lat - north_lat) / (south_lat - north_lat));

// the marker has a height of 40 pixel and a width of 25 pixel
    final BufferedImage logo = ImageIO.read(FileSystems.getDefault().getPath("html").resolve("artwork").resolve("marker-red.png").toFile());
    map.insertBitmap(logo, Math.min(map.getWidth() - 25, Math.max(0, mx - 12)), Math.min(map.getHeight() - 40, Math.max(0, my - 40)), FilterMode.FILTER_ANTIALIASING);

We use the code below to set some text on the map for example the name of the place or the city etc.

map.setDrawMode(DrawMode.MODE_SUB);
map.setColor(0xffffff);
if (text.length() > 0) PrintTool.print(map, 6, 12, 0, uppercase ? text.toUpperCase() : text, -1, false, 100);
PrintTool.print(map, map.getWidth() - 6, map.getHeight() - 6, 0, "MADE WITH LOKLAK.ORG", 1, false, 50);

Finally we draw a marker on map

    int mx = (int) (map.getWidth() * (lon - west_lon) / (east_lon - west_lon));
    int my = (int) (map.getHeight() * (lat - north_lat) / (south_lat - north_lat));
      
    final BufferedImage logo = ImageIO.read(FileSystems.getDefault().getPath("html").resolve("artwork").resolve("marker-red.png").toFile());
    map.insertBitmap(logo, Math.min(map.getWidth() - 25, Math.max(0, mx - 12)), Math.min(map.getHeight() - 40, Math.max(0, my - 40)), FilterMode.FILTER_ANTIALIASING);

  At last we set the copyright message of OPENSTREETMAP at the bottom left.

PrintTool.print(map, 6, map.getHeight() - 6, 0, "(C) OPENSTREETMAP CONTRIBUTORS", -1, false, 100);

At the end we write the image and set the headers for Cross Origin Access.

    response.addHeader("Access-Control-Allow-Origin", "*");
        RemoteAccess.writeImage(fileType, response, post, map);
        post.finalize();
    }
}

 The servlet can be tested locally at

http://localhost:4000/vis/map.png?text=Test&mlat=1.28373&mlon=103.84379&zoom=18

Or on the live SUSI Server
http://api.susi.ai/vis/map.png?text=Test&mlat=1.28373&mlon=103.84379&zoom=18

Output:

References:

https://www.openstreetmap.org/

http://wiki.openstreetmap.org/wiki/Java_Access_Example

https://github.com/fossasia/susi_server/

Continue ReadingMap Responses from SUSI Server

Setting up Your Own Custom SUSI AI server

When you chat with any of the SUSI clients, all the requests are made to standard SUSI server. But let us say that you have implemented some features on server and want to test it. Simply launch your server and you get your own SUSI server. You can then change the option to use your own server on the SUSI Android and Web Chat client.

The first step to get your own copy of the SUSI server is browse to https://github.com/fossasia/susi_server and fork it. Clone your origin head to your local machine and make the changes that you want. There are various tutorials on how to add more servlets to SUSI server, how to add new parameters to an existing action type or maybe modification or a similar type of memory servlet.

To start coding, you can either use some IDE like IDEA by Intelij (download it from here) or open the files you want to modify in a text editor. To start the SUSI server via command line terminal (and not IDE) do the following:

 

Open the terminal in the project directory cloned. Now write in the following command (It is expected that you have your JAVA installed with JAVA_PATH variable setup):

./gradlew build

This will install all the project dependencies required for the support of the project. Next execute :

bin/start.sh

This will take a little time but you will soon see a popped up window in your default browser and the server will start. The landing page of the website will launch. Make the modifications in the server you were aiming. The server is up now. You can access it at local ip address here:

http://127.0.0.1/

But if you try to add this local address in the URL field of any client, it will not work and give you an error. To access this server using a client, open your terminal and execute

ipconfig	//if you are running a windows machine{not recommended}
ifconfig	//if you are running a linux machine{recommended for better performance}

 

This will give you a list of various IP. Find Wireless LAN adapter Wi-Fi. Copy the IPv4 IP address. This is the link you need to enter in the server URL field in the client (Do not forget to add http:// before your IP address). Now sign up and you will be good to go.

Additional Resources :

Continue ReadingSetting up Your Own Custom SUSI AI server

Implementing Speech To Text in SUSI iOS

SUSI being an intelligent bot has the capabilities by which the user can provide input in a hands-free mode by talking and not requiring to even lift the phone for typing. The speech to text feature is available in SUSI iOS with the help of the Speech framework which was released alongside iOS 10 which enables continuous speech detection and transcription. The detection is really fast and supports around 50 languages and dialects from Arabic to Vietnamese. The speech recognition API does its heavy tasks of detection on Apple’s servers which requires an internet connection. The same API is also not always available on all newer devices and also provides the ability to check if a specific language is supported at a particular time.

How to use the Speech to Text feature?

  • Go to the view controller and import the speech framework
  • Now, because the speech is transmitted over the internet and uses Apple’s servers for computation, we need to ask the user for permissions to use the microphone and speech recognition feature. Add the following two keys to the Info.plist file which displays alerts asking user permission to use speech recognition and for accessing the microphone. Add a specific sentence for each key string which will be displayed to the user in the alerts.
    1. NSSpeechRecognitionUsageDescription
    2. NSMicrophoneUsageDescription

The prompts appear automatically when the functionality is used in the app. Since we already have the Hot word recognition enabled, the microphone alert would show up automatically after login and the speech one shows after the microphone button is tapped.

3) To request the user for authorization for Speech Recognition, we use the method SFSpeechRecognizer.requestAuthorization.

func configureSpeechRecognizer() {
        speechRecognizer?.delegate = self

        SFSpeechRecognizer.requestAuthorization { (authStatus) in
            var isEnabled = false

            switch authStatus {
            case .authorized:
                print("Autorized speech")
                isEnabled = true
            case .denied:
                print("Denied speech")
                isEnabled = false
            case .restricted:
                print("speech restricted")
                isEnabled = false
            case .notDetermined:
                print("not determined")
                isEnabled = false
            }

            OperationQueue.main.addOperation {

                // handle button enable/disable

                self.sendButton.tag = isEnabled ? 0 : 1

                self.addTargetSendButton()
            }
        }
    }

4)   Now, we create instances of the AVAudioEngine, SFSpeechRecognizer, SFSpeechAudioBufferRecognitionRequest,SFSpeechRecognitionTask

let speechRecognizer = SFSpeechRecognizer(locale: Locale.init(identifier: "en-US"))
var recognitionRequest: SFSpeechAudioBufferRecognitionRequest?
var recognitionTask: SFSpeechRecognitionTask?
let audioEngine = AVAudioEngine()

5)  Create a method called `readAndRecognizeSpeech`. Here, we do all the recognition related stuff. We first check if the recognitionTask is running or not and if it does we cancel the task.

if recognitionTask != nil {
  recognitionTask?.cancel()
  recognitionTask = nil
}

6)  Now, create an instance of AVAudioSession to prepare the audio recording where we set the category of the session as recording, the mode and activate it. Since these might throw an exception, they are added inside the do catch block.

let audioSession = AVAudioSession.sharedInstance()

do {

    try audioSession.setCategory(AVAudioSessionCategoryRecord)

    try audioSession.setMode(AVAudioSessionModeMeasurement)

    try audioSession.setActive(true, with: .notifyOthersOnDeactivation)

} catch {

    print("audioSession properties weren't set because of an error.")

}

7)  Instantiate the recognitionRequest.

recognitionRequest = SFSpeechAudioBufferRecognitionRequest()

8) Check if the device has an audio input else throw an error.

guard let inputNode = audioEngine.inputNode else {

fatalError("Audio engine has no input node")

}

9)  Enable recognitionRequest to report partial results and start the recognitionTask.

recognitionRequest.shouldReportPartialResults = true

recognitionTask = speechRecognizer?.recognitionTask(with: recognitionRequest, resultHandler: { (result, error) in

  var isFinal = false // to indicate if final result

  if result != nil {

    self.inputTextView.text = result?.bestTranscription.formattedString

    isFinal = (result?.isFinal)!

  }

  if error != nil || isFinal {

    self.audioEngine.stop()

    inputNode.removeTap(onBus: 0)

    self.recognitionRequest = nil

    self.recognitionTask = nil

  }
})

10) Next, we start with writing the method that performs the actual speech recognition. This will record and process the speech continuously.

  • First, we create a singleton for the incoming audio using .inputNode
  • .installTap configures the node and sets up the buffer size and the format
let recordingFormat = inputNode.outputFormat(forBus: 0)

inputNode.installTap(onBus: 0, bufferSize: 1024, format: recordingFormat) { (buffer, _) in

    self.recognitionRequest?.append(buffer)

}

11)  Next, we prepare and start the audio engine.

audioEngine.prepare()

do {

  try audioEngine.start()

} catch {

  print("audioEngine couldn't start because of an error.")

}

12)  Create a method that stops the Speech recognition.

func stopSTT() {

    print("audioEngine stopped")

    audioEngine.inputNode?.removeTap(onBus: 0)

    audioEngine.stop()

    recognitionRequest?.endAudio()

    indicatorView.removeFromSuperview()



    if inputTextView.text.isEmpty {

        self.sendButton.setImage(UIImage(named: ControllerConstants.mic), for: .normal)

    } else {

        self.sendButton.setImage(UIImage(named: ControllerConstants.send), for: .normal)

    }

        self.inputTextView.isUserInteractionEnabled = true
}

13)  Update the view when the speech recognition is running indicating the user its status. Add below code just below audio engine preparation.

// Listening indicator swift

self.indicatorView.frame = self.sendButton.frame

self.indicatorView.isUserInteractionEnabled = true

let gesture: UITapGestureRecognizer = UITapGestureRecognizer(target: self, action: #selector(startSTT))

gesture.numberOfTapsRequired = 1

self.indicatorView.addGestureRecognizer(gesture)
self.sendButton.setImage(UIImage(), for: .normal)

indicatorView.startAnimating()

self.sendButton.addSubview(indicatorView)

self.sendButton.addConstraintsWithFormat(format: "V:|[v0(24)]|", views: indicatorView)

self.sendButton.addConstraintsWithFormat(format: "H:|[v0(24)]|", views: indicatorView)

self.inputTextView.isUserInteractionEnabled = false

The screenshot of the implementation is below:

       

References

Continue ReadingImplementing Speech To Text in SUSI iOS

How to make a SUSI chat bot skill for Cortana

Cortana is assistant from Microsoft just like Siri from Apple. We can make a skill for Cortana i.e Creating your own bot with in cortana so that you can use your bot through cortana by activating it using invocation name. To create SUSI cortana skill we will use SUSI API and Microsoft bot framework. First of all we will have to make a bot on microsoft bot framework and to do so follow this tutorial. Change code in this tutorial with code given below.

var restify = require('restify');
var builder = require('botbuilder');
var request = require('request');
var http = require('http');

// Setup Restify Server
var server = restify.createServer();
server.listen(process.env.port || process.env.PORT || 8080, function() {
   console.log('listening to ', server.name, server.url);
});
// Create chat bot
var connector = new builder.ChatConnector({
   appId: process.env.appId,
   appPassword: process.env.appPassword
});

setInterval(function() {
       http.get(process.env.HerokuURL);
   }, 1200000);

var bot = new builder.UniversalBot(connector);
server.post('/api/messages', connector.listen());

//getting response from SUSI API upon receiving messages from User
bot.dialog('/', function(session) {
   var msg = session.message.text;
   var options = {
       method: 'GET',
       url: 'http://api.asksusi.com/susi/chat.json',
       qs: {
           timezoneOffset: '-330',
           q: session.message.text
       }
   };
//sending request to SUSI API for response
   request(options, function(error, response, body) {
       if (error) throw new Error(error);
       var ans = (JSON.parse(body)).answers[0].actions[0].expression;
       //responding back to user
       session.say(ans,ans);

   })
});

After making bot we have to configure this bot for Cortana and to do so select cortana from list of channel on your bot from https://dev.botframework.com and add following detail shown in figure below.

Invocation name is name which you will call to use your bot. There are many ways to in invoke your skill like Run <invocation name> , Launch <invocation name>. Here is the complete list of invocation commands. How SUSI skill in cortana actually works is when user invoke SUSI skill cortana acts as middleware to send and receive responses from skill to user. After configuring your bot for Cortana test it with cortana with same hotmail account which you have used for bot framework by calling invocation name. Here is the demo video for testing SUSI Skill https://youtu.be/40yX16hxcls.

You have complete creating a skill for cortana now if you want to publish it to world select manage cortana dashboard from your bots in https://dev.botframework.com and publish it to first by filling form.

If you want to learn more about bot framework refer to https://docs.microsoft.com/en-us/Bot-Framework/index

References:
SUSI Cortana Repository: https://github.com/fossasia/susi_cortana_skill
Tutorial for bot: https://blog.fossasia.org/susi-ai-bots-with-microsofts-bot-framework/

Continue ReadingHow to make a SUSI chat bot skill for Cortana

How to make SUSI kik bot

To make SUSI kik bot first you have to configure a bot. To configure bot go to https://dev.kik.com/ and make your bot by scanning code from your kik mobile app.You have to answer following questions from botsworth and make your bot.

 

After logging in to your dashboard get your api key by going to configuration menu on top.

After your bot is setup follow given steps to create your first susi kik bot.

Steps:

  1. Install Node.js from the link below on your computer if you haven’t installed it already.
    https://nodejs.org/en/
  2. Create a folder with any name and open shell and change your current directory to the new folder you created.
  3. Type npm init in command line and enter details like name, version and entry point.
  4. Create a file with the same name that you wrote in entry point in above given step. i.e index.js and it should be in same folder you created.
  5. Type following commands in command line  npm install –save @kikinteractive/kik. After @kikinteractive/kik is installed type npm install –save http after http is installed type npm install –save request when all the modules are installed check your package.json these modules will be included within dependencies portion.

  6. Your package.json file should look like this.

     
    {
     "name": "susi_kikbot",
       "version": "1.0.0",
     "description": "susi kik bot",
     "main": "index.js",
     "scripts": {
       "test": "node tests/sample.js"
     },
     "license": "MIT",
     "dependencies": {
       "@kikinteractive/kik": "^2.0.11",
       "request": "^2.75.0"
     }
    }
    
    
  7. Copy following code into file you created i.e index.js and add your bot name to it in place of username.

    var http = require('http');
    var Bot = require('@kikinteractive/kik');
    var request = require('request')
    var answer;
     
    var bot = new Bot({
     
        username: '<your-bot-name>',
        apiKey: process.env.API_KEY,
        baseUrl: process.env.HEROKU_URL
     
    });
     
    bot.updateBotConfiguration();
     
    bot.onTextMessage((message) => {
     
        request('http://api.asksusi.com/susi/chat.json?q=' + encodeURI(query), function(error, response, body) {
     
            if (!error && response.statusCode == 200) {
     
                answer = JSON.parse(body).answers[0].actions[0].expression;
     
            } else {
     
                answer = "Oops, Looks like Susi is taking a
                break, She will be back soon";
     
            }
     
        });
     
        message.reply(answer)
    });
     
    http.createServer(bot.incoming()).listen(process.env.PORT || 5000)
    
    
  8. Before deploying our bot to heroku so that it can be active we have to make a github repository for chatbot to make github repository follow these steps.In shell change current directory to folder we created above and  write
    git init
    git add .
    git commit -m”initial”
    git remote add origin <URL for remote repository>
    git remote -v
    git push -u origin master

    You will get URL for remote repository by making repository on your github and copying this link of your repository.

  9. To deploy your bot to heroku you need an account on Heroku and after making an account make an app.

     

  10. Deploy app using github deployment method.
  11. Select Automatic deployment method.
  12. Go to settings of your app and config variables and paste your API key for bot to this and name it as API_KEY and get your heroku app url and make a variable for it named HEROKU_URL.
  13. Your susi bot is ready now test it by massaging it.

If you want to learn more about kik API then refer to https://dev.kik.com/#/docs/messaging

 

Resources:

Github Repository: https://github.com/fossasia/susi_kikbot
KIK bot API: https://dev.kik.com/#/docs/messaging

Continue ReadingHow to make SUSI kik bot

SUSI AI Bots with Microsoft’s Bot Framework

The Bot Framework is used to build intelligent chatbots and it supports .NET, Node.js, and REST. To learn about building bots using bot framework go to  https://docs.microsoft.com/en-us/bot-framework/bot-builder-overview-getstarted . Now to build SUSI AI bot for different platforms like facebook, telegram, kik, skype follow below given steps.

  1. Install Node.js from the link below on your computer if you haven’t installed it already.
    https://nodejs.org/en/
  2. Create a folder with any name and open a shell and change your current directory to the new folder you created.
  3. Type npm init in shell and enter details like name, version and entry point.
  4. Create a file with the same name that you wrote in entry point in above given step. i.e index.js and it should be in same folder you created.
  5. Type following commands in command line  npm install –save restify.After restify is installed type npm install –save botbuilder   after botbuilder is installed type npm install –save request when all the modules are installed check your package.json modules will be included within dependencies portion.

  6. Your package.json file should look like this.

    {
    "name": "skype-bot",
    "version": "1.0.0",
    "description": "SUSI AI Skype Bot",
    "main": "app.js",
    "scripts": {
      "test": "echo \"Error: no test specified\" && exit 1",
      "start": "node app.js"
    },
    "author": "",
    "license": "ISC",
    "dependencies": {
      "botbuilder": "^3.8.1",
      "request": "^2.81.0",
      "restify": "^4.3.0"
    }
    }
    
  7. Copy following code into file you created i.e index.js

    var restify = require('restify');
    var builder = require('botbuilder');
    var request = require('request');
    
    // Setup Restify Server
    var server = restify.createServer();
    server.listen(process.env.port || process.env.PORT || 8080, function() {
       console.log('%s listening to %s', server.name, server.url);
    });
    
    // Create chat bot
    var connector = new builder.ChatConnector({
     appId: process.env.appId,
     appPassword: process.env.appPassword
    });
    
    var bot = new builder.UniversalBot(connector);
    server.post('/api/messages', connector.listen());
    //When bot is added by user
    bot.on('contactRelationUpdate', function(message) {
       if (message.action === 'add') {
           var name = message.user ? message.user.name : null;
           var reply = new builder.Message()
               .address(message.address)
               .text("Hello %s... Thanks for adding me. You can talk to SUSI now.", name || 'there');
           bot.send(reply);
       }
    });
    //getting response from SUSI API upon receiving messages from User
    bot.dialog('/', function(session) {
       var options = {
           method: 'GET',
           url: 'http://api.asksusi.com/susi/chat.json',
           qs: {
               timezoneOffset: '-330',
               q: session.message.text
           }
       };
    //sending request to SUSI API for response 
       request(options, function(error, response, body) {
           if (error) throw new Error(error);
           var ans = (JSON.parse(body)).answers[0].actions[0].expression;
           //responding back to user
           session.send(ans);
    
       })
    });
    
  8. You have to replace appID and appPassword with your own ID and Password which you can get by below given steps.
  9. Sign in/Sign up to this https://dev.botframework.com/. After signing in go to My Bots option at the top of the page and Create/Register your bot. Enter details of your bot and click on “Create Microsoft App ID and password”. 
  10. Leave messaging endpoint for now after getting app ID and password we will write messaging endpoint.
  11. Copy your APP ID and Password and save them for later use. Paste your App ID in box given for ID on bot registration page.

  12. Now we have to create messaging endpoint to listen for requests. Make a github repository and push the files in the folder we created above.

    In command line change current directory to folder we created above and  write
    git init
    git add .
    git commit -m”initial”
    git remote add origin <URL for remote repository>
    git remote -v
    git push -u origin master  You will URL for remote repository by making repository on your github and copying this link of your repository.

  13. Now we have to deploy this github repository to heroku to get url for messaging endpoint. If you don’t have account on heroku sign up here https://www.heroku.com/ else just sign in and create a new app.
  14. Deploy your repository onto heroku from deploy option and choosing github as a deployment method.
  15. Select automatic deployment so that you make any changes in github repository they should be deployed to heroku.

  16. Open you app from option on top right and copy the link of your heroku app and append it with /api/messages and enter this url as messaging endpoint.

    https://{Your_App_Name}.herokuapp.com/api/messages
  17. Register the bot and add APP ID and password you saved to your heroku app in settings->config variables.
  18. Now go to  https://dev.botframework.com/. and then in My Bots go to your bot and click on Skype bot then add it to contact and start chatting.
  19. You can connect same bot to different channels like kik, slack, telegram, facebook and many others.

    Add different channels in your bot page and follow these links for deploying onto different platforms.

If you want to learn more about bot framework you can refer to https://docs.microsoft.com/en-us/Bot-Framework/index

Resources:
Code: https://github.com/fossasia/susi_skypebot
Bot Framework: https://docs.microsoft.com/en-us/bot-framework/bot-builder-overview-getstarted

Continue ReadingSUSI AI Bots with Microsoft’s Bot Framework