Skills for SUSI

Susi is an open source personal assistant can do a lot of amazing things for you apart from just answering queries in the text. Susi supports many action type such as answer, table, pie chart, RSS, web search, map. Actions contain a list of objects each with type attribute, there can be more than one actions in the object list. For example

curl http://api.susi.ai/susi/chat.json?timezoneOffset=-330&q=Who+are+you

We get a json response similar to

{
"query": "Who are you",
"answers": [{
"data": [],
"metadata": {"count": 0},
"actions": [{
"type": "answer",
"expression": "I was told that I am a Social Universal Super Intelligence. What do you think?"
}]
}],
}

The above query is an example of action type ‘answer’, for developing more skills on action type answer refer to the Fossasia  blog : How to teach Susi skills.  In this blog we will see how to teach a table skill to susi and how susi interprets the skill .So let’s add a skill to display football teams and its players using service Football-Data.org .

For writing rules Open a new etherpad with a desired name <etherpad name> at http://dream.susi.ai/

  1. Next let’s set some query for which we want Susi to answer.

Example queries:

tell me the teams in premier league | Premier league teams

To get answer we define the following rule

!console:
{
"url":"http://api.football-data.org/v1/competitions/398/teams",
"path":"$.teams",
"actions":[{
"type":"table",
"columns":{"name":"Name","code":"Code","shortName":"Short Name","crestUrl":Logo},
"count": -1
}]
}
eol

Expalanation:

The JSON response for above URL look like this:

{
"_links": {
"self": {
"href": "http://api.football-data.org/v1/competitions/398/teams"
},
"competition": {
"href": "http://api.football-data.org/v1/competitions/398"
}
},
"count": 20,
"teams": [{
"_links": {
"self": {
"href": "http://api.football-data.org/v1/teams/66"
},
"fixtures": {
"href": "http://api.football-data.org/v1/teams/66/fixtures"
},
"players": {
"href": "http://api.football-data.org/v1/teams/66/players"
}
},
"name": "Manchester United FC",
"code": "MUFC",
"shortName": "ManU",
"squadMarketValue": null,
"crestUrl": "http://upload.wikimedia.org/wikipedia/de/d/da/Manchester_United_FC.svg"
},
{
"_links": {
"self": {
"href": "http://api.football-data.org/v1/teams/65"
},
"fixtures": {
"href": "http://api.football-data.org/v1/teams/65/fixtures"
},
"players": {
"href": "http://api.football-data.org/v1/teams/65/players"
}
},
"name": "Manchester City FC",
"code": "MCFC",
"shortName": "ManCity",
"squadMarketValue": null,
"crestUrl": "https://upload.wikimedia.org/wikipedia/en/e/eb/Manchester_City_FC_badge.svg"
}]
}

The attribute ‘path’  statuses object contains a list of objects for which we want to show the table. The table is defined with the action type “table” and a columns object which provides a mapping from the column value names to the descriptive names that will be rendered in the client’s output. In our case, there are  4 columns  Name of the team, Team Code, Short Name of the team, and the Team logo’s URL.

The count attribute is used to denote how many rows to populate the table. If count = -1 , then it means as many as possible or displays all the results.It’s easy, isn’t it? We have successfully created table skill for SUSI. Let’s try it out

Go to http://chat.susi.ai/ and type: dream < your dream name>. And type your queries like  “tell me the teams in premier league” in our case  and see how SUSI presents you the results


Next, want to create some more skills. Let’s teach SUSI to play a game Rock, Paper, Scissors, Lizard, Spock. SUSI can give random answers to your queries for example

What is your favorite dish
Potatoes|Vegetables|Fish

SUSI will return any random answer “Potatoes, “Vegetables” or “Fish”. We would use this to create a game.
First form a general query like “I am getting bored let us play something” and the add an answer to it by adding the rules of the game something like

I am getting bored let us play something
Let's play the game Rock, Paper, Scissors, Lizard, Spock. It is an expansion to the game Rock, Paper, Scissors. We both will  pick a variable and reveal. The winner will be  one who defeats the others. It goes like Scissors cuts Paper,Paper covers Rock, Rock crushes Lizard, Lizard poisons Spock, Spock smashes Scissors, Scissors decapitates Lizard, Lizard eats Paper, Paper disproves Spock, Spock vaporizes Rock, (and as it always has) Rock crushes Scissors. Let's begin, choose something

Next since user would choose between any 5 of the Rock, Scissors, Lizard,Paper and spock add answers to each of them based on the rules.

Rock | I choose Rock
Paper, Paper covers Rock I won :) | Sccisors, Rock crushes Scissors , I lost :( | Lizard, Rock crushes Lizard , You won :-) | Spock, Spock vaporises Rock, You lost :) | Rock, Ahh It's a Tie :-D

Paper | I choose Paper
Rock, Paper covers Rock You won :( | Sccisors, Scissors cuts Paper I won :) | Lizard, Lizard eats Paper I won :) | Spock , Paper disproves Spock You won :( | Paper, Ahh It's a Tie :-D

Lizard | I choose Lizard
Rock, Rock crushes Lizard , I won :-) | Spock, Lizard poisons Spock You won :( | Sccisors, Scissors decapitates Lizard I won :) | Paper, Lizard eats Paper You won :( | Lizard, Ahh It's a Tie :-D

Scissors | I choose Scissors
Rock, Rock crushes Scissors , You lost :) | Paper , Scissors cuts Paper You won :( | Spock, Spock smashes Scissors I won :) | Lizard, Scissors decapitates Lizard You won :( | Scissors, Ahh It's a Tie :-D

Spock | I choose Spock
Rock, Spock vaporizes Rock, I lost :( | Lizard, Lizard poisons Spock You lost :) | Scissors, Spock smashes Scissors I lost :(| Paper, Paper disproves Spock I won :) | Spock, Ahh It's a Tie :-D

SUSI will return any random answer to the variable picked go and try to beat SUSI.

Have fun with SUSI and do create some other awesome skills for SUSI, read this tutorial  for more details.

Continue ReadingSkills for SUSI

Hotword Detection with Pocketsphinx for SUSI.AI

Susi has many apps across all the major platforms. Latest addition to them is the Susi Hardware which allows you to setup Susi on a Hardware Device like Raspberry Pi.

Susi Hardware was able to interact with a push of a button, but it is always cool and convenient to call your assistant anytime rather than using a button.

Hotword Detection helps achieve that. Hotword Detection involves running a process in the background that continuously listens for voice. On noticing an utterance, we need to check whether it contains desired word. Hotword Detection and its integration with Susi AI can be explained using the diagram below:

 

What is PocketSphinx?

PocketSphinx is a lightweight speech recognition engine, specifically tuned for handheld and mobile devices, though it works equally well on the desktop.

PocketSphinx is free and open source software. PocketSphinx has various applications but we utilize its power to detect a keyword (say Hotword) in a verbally spoken phrase.

Official Github Repository: https://github.com/cmusphinx/pocketsphinx

Installing PocketSphinx

We shall be using PocketSphinx with Python. Latest version on it can be installed by

pip install pocketsphinx

If you are using a Raspberry Pi or ARM based other board, Python 3.6 , it will install from sources by the above step since author doesn’t provide a Python Wheel. For that, you may need to install swig additionally.

sudo apt install swig

How to detect Hotword with PocketSphinx?

PocketSphinx can be used in various languages. For Susi Hardware, I am using
Python 3.

Steps:

      • Import PyAudio and PocketSphinx
        from pocketsphinx import *
        import pyaudio
        

         

      • Create a decoder with certain model, we are using en-us model and english us default dictionary. Specify a keyphrase for your application, for Susi AI , we are using “Susi” as Hotword
        pocketsphinx_dir = os.path.dirname(pocketsphinx.__file__)
        model_dir = os.path.join(pocketsphinx_dir, 'model')
        
        config = pocketsphinx.Decoder.default_config()
        config.set_string('-hmm', os.path.join(model_dir, 'en-us'))
        config.set_string('-keyphrase', 'susi')
        config.set_string('-dict', os.path.join(model_dir, dict_name))
        config.set_float('-kws_threshold', self.threshold)

         

      • Start a PyAudio Stream from Microphone Input
        p = pyaudio.PyAudio()
        
        stream = p.open(format=pyaudio.paInt16, channels=1, rate=16000, input=True, frames_per_buffer=20480)
        stream.start_stream()

         

      • In a forever running loop, read from stream and process buffer in chunks of 1024 frames if it is not empty.
        buf = stream.read(1024)
        if buf:
            decoder.process_raw(buff)
        else:
            break;
        

         

      • Check for hotword and start speech recognition if hotword detected. After returning from the method, start detection again.

        if decoder.hyp() is not None:
            print("Hotword Detected")
            decoder.end_utt()
            start_speech_recognition()
            decoder.start_utt()
        

         

      • Run the app, if detection doesn’t seem to work well, adjust kws_threshold in step 2 to give optimal results.

In this way, Hotword Detection can be added to your Python Project. You may also develop some cool hacks with our AI powered Assistant Susi by Voice Control.
Check repository for more info: https://github.com/fossasia/susi_hardware

Continue ReadingHotword Detection with Pocketsphinx for SUSI.AI

Displaying Web Search and Map Preview using Glide in SUSI Android App

SUSI.AI has many skills. Two of which are displaying web search of a certain query and displaying a map of certain position. This post will cover how these things are implemented in Susi Android App and how a preview is displayed using Bumptech’s free and open source Glide library. So, what is glide? According to Glide docs, Glide is a fast and efficient open source media management and image loading framework for Android that wraps media       decoding, memory and disk caching, and resource pooling into a simple and easy to use interface.

Now, lets describe how this framework is used in Susi App. So, let’s cover it’s two uses one by one :

Map Preview :

    

Whenever a user queries something like “Where is Singapore” or “Where am I”, the response from server is a map with latitude, longitude and zoom level. See the “type” which is map. It indicates android app that it needs to display a map with provided values.

“actions”: [
     {
       “type”: “answer”,
       “expression”: “Singapore is a place with a population of 3547809. Here is a map: https://www.openstreetmap.org/#map=13/1.2896698812440377/103.85006683126556”
     },
     {
       “type”: “anchor”,
       “link”: “https://www.openstreetmap.org/#map=13/1.2896698812440377/103.85006683126556”,
       “text”: “Link to Openstreetmap: Singapore”
     },
     {
       “type”: “map”,
       “latitude”: “1.2896698812440377”,
       “longitude”: “103.85006683126556”,
       “zoom”: “13”
     }
   ]
 }]

So, these values are then plugged into the below mentioned url where length x width is size of image to be retrieved. This url links to an image of the map with center and size as defined.

http://staticmap.openstreetmap.de/staticmap.php?center=latitude,longitude&zoom=zoom&size=lengthxwidth

So, now as we have a link to the image to be displayed. We just need something to get the image from that link and display it in the app. That’s where Glide comes to rescue. It loads the image from the link to an ImageView.

Glide.with(currContext).load(mapHelper.getMapURL()).listener(new RequestListener<String, GlideDrawable>() {

@Override
public boolean onException(Exception e, String model, Target<GlideDrawable> target, boolean isFirstResource) {
return false;
}

@Override
public boolean onResourceReady(GlideDrawable resource, String model,
Target<GlideDrawable> target, boolean isFromMemoryCache, boolean isFirstResource){
mapViewHolder.pointer.setVisibility(View.VISIBLE);
return false;
}

}).into(mapViewHolder.mapImage);

So, what the above code does is that it displays image from url mapHelper.getMapURL() to mapViewHolder.mapImage.

So, by this way, we are able to display a preview of the map in the chat itself. The user can then click on the image to load the         complete map.

Web Search Preview :

When a user enters queries like “search for dog”, the response from server is a websearch with the query to be searched,         something like this.

“actions”: [
     {
       “type”: “answer”,
       “expression”: “Here is a web search result:”
     },
     {
       “type”: “websearch”,
       “query”: “dog”
     }
   ]

So, now we know the query to be searched, we can use any search api to get results about that query ad display it in the app. In Susi android app, we have used DuckDuckGo open source api to do that task. We call this url

https://api.duckduckgo.com/?format=json&pretty=1&q=query&ia=meanings

which gives a json response with results. We then use the json     results which contains the link to result, image to be displayed and a short abstract of the result.

{
“Result” : “<a href=\”https://duckduckgo.com/Dog\”>Dog</a>A member of genus Canis that forms part of the wolf-like canids, and is the most widely abundant…”,
“Icon” : {
“URL” : “https://duckduckgo.com/i/16101b42.jpg”,
“Height” : “”,
“Width” : “”
},
“FirstURL” : “https://duckduckgo.com/Dog”,
“Text” : “Dog A member of genus Canis that forms part of the wolf-like canids, and is the most widely abundant…”
}

There are three things we are displaying in the websearch preview, which are Title, text and an image. We display the query as the title, text in json response as text and use glide to get the image from the url provided in the the json response.

glide.load(url)
.crossFade()
.into(websearchholder.previewImageView);

Using the above code, we load the image from the image url to websearchholder.previewImageView().

So, by this way, we are able to display a preview of the search     result. The user can then touch on this preview which opens the     detailed page of the result.

Happy Coding!

Continue ReadingDisplaying Web Search and Map Preview using Glide in SUSI Android App

Conversion of CSS styles into React styles in SUSI Web Chat App

Earlier this week we had an issue where the text in our search box of the SUSI web app was not white even after having all the required styles. After careful inspection it was found that there is a attribute named -webkit-text-fill-color which was set to black.

Now I faced this issue as adding such attribute to our reactJs code will cause lint errors. So after careful searching stackoverflow, i found a way to add css attribute to our react code by using different case. I decided to write a blog on this for future reference and it might come handy to other developers as well.

If you want to write css in javascript, you have to turn dashed-key-words into camelCaseKeys

For example:

background-color => backgroundColor
border-radius => borderRadius
but vendor prefix starts with capital letter (except ms)
-webkit-box-shadow => WebkitBoxShadow (capital W)
-ms-transition => msTransition ('ms' is the only lowercase vendor prefix)

const containerStyle = {
  WebkitBoxShadow: '0 0 0 1000px white inset'
};

So in our case:-

-webkit-text-fill-color became WebkitTextFillColor

The final code of styles looked like: –

const searchstyle = {
      WebkitTextFillColor: 'white',
      color: 'white'
    }

Now, because inline styles gets attached on tags directly instead of using selectors, we have to put this style on the <input> tag itself, not the container.

See the react doc #inline-styles section for more details.

Continue ReadingConversion of CSS styles into React styles in SUSI Web Chat App

How to add a new attribute in an action type for SUSI.AI

In this blog, I’ll be telling on how to add more attributes to already implemented action types so that it becomes easy to filter out the JSON results that we are getting by calling various APIs.

What are actions?

Actions are a set of defined functionality that how Susi works for special JSON return type from some API endpoint. These include table, pie chart, rss, web search etc. These action types are implemented and defined at the backend(on server). The definition contains several attributes and their data types.

Current action types and their attributes:

  • table : {columns, length}
  • piechart : {total, key, value, unit}
  • rss : (title, description, link)
  • websearch : {query}
  • map : {latitude, longitude, zoom}

* Please see that these are as of the date of publish of this blog. These are subject to change. Also keep in mind that these are optional attributes.

Need for new attributes:

There are millions of open APIs out there and hence their JSON response structure also varies. Some return a JSON Array and some return a single JSON object. Some might return JSON Array inside a JSONObject. They all varies. So to give proper filters, we may need new action types.

In this blog, we’ll take up the example of how “length” attribute was added. Similarly you can add more attributes to any of the action types.

Some APIs return few as 5 elements in a JSON Array and some give out 100s of elements. So the power of limiting the rows in a particular skill is given to Susi-skill developers only with addition of length parameter.

First things first.. Identify the action type in which you want to add parameters. Browse to SusiMind.java and SusiAction.java files. These are the main files where Susi looks for attributes while evaluating  a dream or a skill.

In SusiAction.java file, find the corresponding public action method which is returning a JSONObject. Currently the method looks like this:

public static JSONObject tableAction(JSONObject cols) {
        JSONObject json = new JSONObject(true)
            .put("type", RenderType.table.name())
            .put("columns", cols);
        return json;
    }

Add a parameter of corresponding data type (attribute you are adding) in argument list. Following this, put that variable in in JSONObject variable (that is json) which also being returned at the end. After changes, the method will look like this:

public static JSONObject tableAction(JSONObject cols, int length) {
        JSONObject json = new JSONObject(true)
            .put("type", RenderType.table.name())
            .put("columns", cols)
            .put("length", length);
        return json;
    }

In SusiMind.java file, find the if condition where other attributes of the corresponding action type are being checked. (sticking to length attribute in table action type, following is what you have to do).

if (type.equals(SusiAction.RenderType.table.toString()) && boa.has("columns")) {                                                                     actions.put(SusiAction.tableAction(boa.getJSONObject("columns"));
}

Now depending on the need, add the conditions in the code, make the changes. The demand of this action type is : if columns attribute is present, then the user may or may not put the length parameter, but if the columns are absent, length will have no value. So now the code gets modified and looks like this :

if (type.equals(SusiAction.RenderType.table.toString()) && boa.has("columns")) {                                                                                                      actions.put(SusiAction.tableAction(boa.getJSONObject("columns"),
         		boa.has("length") ?boa.getInt("length") : -1));
}

But wait.. To see these changes up on Susi, open an issue on the server repo and on all the client repos as well.

Server | ios client | android client | web chat client

Once you are sure things are working as expected, send a pull request to server after making the changes(follow the pull request practices followed at FOSSASIA).

 

Continue ReadingHow to add a new attribute in an action type for SUSI.AI

Youtube videos in the SUSI iOS Client

The iOS and android client already have the functionality to play videos based on the queries. In order to implement this feature of playing videos in the iOS client, we use the Youtube Data API v3. The task here was to create an UI/UX for the playing of videos within the app. An API call is made initially to fetch the youtube videos based on the query and they video ID of the first object is extracted and used to play the video.

The API endpoint for youtube data API looks like:

https://www.googleapis.com/youtube/v3/search?part=snippet&q={query}&key={your_api_key}

Using this we get the following result: ( I am adding only the first item which is required since the response is too long )

Path: $.items[0]

{

 "kind": "youtube#searchResult",

 "etag": "\"m2yskBQFythfE4irbTIeOgYYfBU/oR-eA572vNoma1XIhrbsFTotfTY\"",

 "id": {

"kind": "youtube#channel",

"channelId": "UCQprMsG-raCIMlBudm20iLQ"

 },

 "snippet": {

"publishedAt": "2015-01-01T11:06:00.000Z",

"channelId": "UCQprMsG-raCIMlBudm20iLQ",

"title": "FOSSASIA",

"description": "FOSSASIA is supporting the development of Free and Open Source technologies for social change in Asia. The annual FOSSASIA Summit brings together ...",

"thumbnails": {

"default": {

"url": "https://yt3.ggpht.com/-CP18cWbo34A/AAAAAAAAAAI/AAAAAAAAAAA/kEmIgO8OjCk/s88-c-k-no-mo-rj-c0xffffff/photo.jpg"

},

"medium": {

"url": "https://yt3.ggpht.com/-CP18cWbo34A/AAAAAAAAAAI/AAAAAAAAAAA/kEmIgO8OjCk/s240-c-k-no-mo-rj-c0xffffff/photo.jpg"

},

"high": {

"url": "https://yt3.ggpht.com/-CP18cWbo34A/AAAAAAAAAAI/AAAAAAAAAAA/kEmIgO8OjCk/s240-c-k-no-mo-rj-c0xffffff/photo.jpg"

}

},

"channelTitle": "FOSSASIA",

"liveBroadcastContent": "upcoming"

 }

}

We parse the above object to grab the videoID, based on the query,  we will use code below:

if let itemsObject = response[Client.YoutubeResponseKeys.Items] as? [[String : AnyObject]] {
    if let items = itemsObject[0][Client.YoutubeResponseKeys.ID] as? [String : AnyObject] {
         let videoID = items[Client.YoutubeResponseKeys.VideoID] as? String
         completion(videoID, true, nil)
    }
}

This videoID is returned to the Controller where this method was called.

Now, we begin with designing the UI for the same. First of all, we need a view on which the youtube video will be played and this view would help dismiss the video by clicking on it.

First, we add the blackView to the entire screen.

// declaration
let blackView = UIView()

// Add backgroundView
func addBackgroundView() {

   If let window = UIApplication.shared.keyWindow {

           self.view.addSubview(blackView) 

           // Cover the entire screen
           blackView.frame = window.frame

           blackView.backgroundColor = UIColor(white: 0, alpha: 0.5)
           blackView.addGestureRecognizer(UITapGestureRecognizer(target: self, action: #selector(handleDismiss)))

   }

}

func handleDismiss() {
   UIView.animate(withDuration: 0.5, delay: 0, usingSpringWithDamping: 1, initialSpringVelocity: 1, options: .curveEaseOut, animations: {
       self.blackView.removeFromSuperview()
   }, completion: nil)
}

Next, we add the YoutubePlayerView. For this we use the Pod `YoutubePlayer`. Since, it had a few warnings showing as well as some videos not being played I had to make fixes to the original pod and use my own customized version ( available here ).

// Youtube Player
lazy var youtubePlayer: YouTubePlayerView = {
    let frame = CGRect(x: 0, y: 0, width: self.view.frame.width - 16, height: self.view.frame.height * 1 / 3)
    let player = YouTubePlayerView(frame: frame)
    return player
}()

// Shows Youtube Player

func addYotubePlayer(_ videoID: String) {
    if let window = UIApplication.shared.keyWindow {

       // Add YoutubePlayer view on top of blackView
        self.blackView.addSubview(self.youtubePlayer)
        // Calculate and set frame

       let centerX = UIScreen.main.bounds.size.width / 2
        let centerY = UIScreen.main.bounds.size.height / 3
        self.youtubePlayer.center = CGPoint(x: centerX, y: centerY)

       // Load Player using the Video ID 
        self.youtubePlayer.loadVideoID(videoID)

        blackView.alpha = 0
        youtubePlayer.alpha = 0

        UIView.animate(withDuration: 0.5, delay: 0, usingSpringWithDamping: 1, initialSpringVelocity: 1, options: .curveEaseOut, animations: {
            self.blackView.alpha = 1
            self.youtubePlayer.alpha = 1
       }, completion: nil)
    }
}

We are set with the UI and the only thing we are left with is to actually call the API in the client and after getting the `videoID` from that we call the above method passing this `videoID`. Before calling we need to check whether our query contains the play action or not and if it does we make the API call and add the player.

if let text = inputTextField.text {
    if text.contains("play") || text.contains("Play") {
        let query = text.replacingOccurrences(of: "play", with: "").replacingOccurrences(of: "Play", with: "")
        Client.sharedInstance.searchYotubeVideos(query) { (videoID, _, _) in
            DispatchQueue.main.async {
                if let videoID = videoID {
                    self.addYotubePlayer(videoID)
                 }
             }
          }
    }

}

We are all set now!Below is the output for the Youtube Player:

Continue ReadingYoutube videos in the SUSI iOS Client

Websearch and Link Preview support in SUSI iOS

The SUSI.AI server responds to API calls with answers to the queries made. These answers might contain an action, for example a web search, where the client needs to make a web search request to fetch different web pages based on the query. Thus, we need to add a link preview in the iOS Client for each such page extracting and displaying the title, description and a main image of the webpage.

At first we make the API call adding the query to the query parameter and get the result from it.

API Call:

http://api.susi.ai/susi/chat.json?timezoneOffset=-330&q=amazon

And get the following result:

{

"query": "amazon",

"count": 1,

"client_id": "aG9zdF8xMDguMTYyLjI0Ni43OQ==",

"query_date": "2017-06-02T14:34:15.675Z",

"answers": [

{

"data": [{

"0": "amazon",

"1": "amazon",

"timezoneOffset": "-330"

}],

"metadata": {

"count": 1

},

"actions": [{

"type": "answer",

"expression": "I don't know how to answer this. Here is a web search result:"

},

{

"type": "websearch",

"query": "amazon"

}]

}],

"answer_date": "2017-06-02T14:34:15.773Z",

"answer_time": 98,

"language": "en",

"session": {

"identity": {

"type": "host",

"name": "108.162.246.79",

"anonymous": true

}

}

}

After parsing this response, we first recognise the type of action that needs to be performed, here we get `websearch` which means we need to make a web search for the query. Here, we use `DuckDuckGo’s` API to get the result.

API Call to DuckDuckGo:

http://api.duckduckgo.com/?q=amazon&format=json

I am adding just the first object of the required data since the API response is too long.

Path: $.RelatedTopics[0]

{

 "Result": "<a href=\"https://duckduckgo.com/Amazon.com\">Amazon.com</a>Amazon.com, also called Amazon, is an American electronic commerce and cloud computing company...",

 "Icon": {

"URL": "https://duckduckgo.com/i/d404ba24.png",

"Height": "",

"Width": ""

},

 "FirstURL": "https://duckduckgo.com/Amazon.com",

 "Text": "Amazon.com Amazon.com, also called Amazon, is an American electronic commerce and cloud computing company..."

}

For the link preview, we need an image logo, URL and a description for the same so, here we will use the `Icon.URL` and `Text` key. We have our own class to parse this data into an object.

class WebsearchResult: NSObject {
   
   var image: String = "no-image"
   var info: String = "No data found"
   var url: String = "https://duckduckgo.com/"
   var query: String = ""
   
   init(dictionary: [String:AnyObject]) {
       
       if let relatedTopics = dictionary[Client.WebsearchKeys.RelatedTopics] as? [[String : AnyObject]] {
           
           if let icon = relatedTopics[0][Client.WebsearchKeys.Icon] as? [String : String] {
               if let image = icon[Client.WebsearchKeys.Icon] {
                   self.image = image
               }
           }
           
           if let url = relatedTopics[0][Client.WebsearchKeys.FirstURL] as? String {
               self.url = url
           }
           
           if let info = relatedTopics[0][Client.WebsearchKeys.Text] as? String {
               self.info = info
           }
           
           if let query = dictionary[Client.WebsearchKeys.Heading] as? String {
               let string = query.lowercased().replacingOccurrences(of: " ", with: "+")
               self.query = string
           }   
       }   
   }   
}

We now have the data and only thing left is to display it in the UI.

Within the Chat Bubble, we need to add a container view which will contain the image, and the text description.

 let websearchContentView = UIView()
    
    let searchImageView: UIImageView = {
        let imageView = UIImageView(frame: CGRect(x: 0, y: 0, width: 44, height: 44))
        imageView.contentMode = .scaleAspectFit

       // Placeholder image assigned
        imageView.image = UIImage(named: "no-image")
        return imageView
    }()
    
    let websiteText: UILabel = {
        let label = UILabel()
        label.textColor = .white
        return label
    }()

   func addLinkPreview(_ frame: CGRect) {
        textBubbleView.addSubview(websearchContentView)
        websearchContentView.backgroundColor = .lightGray
        websearchContentView.frame = frame
        
        websearchContentView.addSubview(searchImageView)
        websearchContentView.addSubview(websiteText)




       // Add constraints in UI
        websearchContentView.addConstraintsWithFormat(format: "H:|-4-[v0(44)]-4-[v1]-4-|", views: searchImageView, websiteText)
        websearchContentView.addConstraintsWithFormat(format: "V:|-4-[v0]-4-|", views: searchImageView)
        websearchContentView.addConstraintsWithFormat(format: "V:|-4-[v0(44)]-4-|", views: websiteText)
    }

Next, in the Collection View, while checking other action types, we add checking for `websearch` and then call the API there followed by adding frame size and calling the `addLinkPreview` function.

else if message.responseType == Message.ResponseTypes.websearch {
  let params = [
    Client.WebsearchKeys.Query: message.query!,
    Client.WebsearchKeys.Format: "json"
  ]

  Client.sharedInstance.websearch(params, { (results, success, error) in                      
    if success {
      cell.message?.websearchData = results
      message.websearchData = results
      self.collectionView?.reloadData()
      self.scrollToLast()
    } else {
      print(error)
    }
  })
                    
  cell.messageTextView.frame = CGRect(x: 16, y: 0, width: estimatedFrame.width + 16, height: estimatedFrame.height + 30)
  

 cell.textBubbleView.frame = CGRect(x: 4, y: -4, width: estimatedFrame.width + 16 + 8 + 16, height: estimatedFrame.height + 20 + 6 + 64)
                    
  let frame = CGRect(x: 16, y: estimatedFrame.height + 20, width: estimatedFrame.width + 16 - 4, height: 60 - 8)
  

 cell.addLinkPreview(frame)

}

And set the collection View cell’s size.

else if message.responseType == Message.ResponseTypes.websearch {
  return CGSize(width: view.frame.width, height: estimatedFrame.height + 20 + 64)
}

And we are done 🙂

Here is the final version how this would look like on the device:

 

Continue ReadingWebsearch and Link Preview support in SUSI iOS

Setup Lint Testing in SUSI Android

As developers tend to make mistakes while writing code, small mistakes or issues can cause a negative impact on the overall functionality and speed of the app. Therefore it is necessary to understand the importance of Lint testing in Android.

Android Lint is a tool present in Android studio which is effective in scanning and reporting different types of bugs present in the code, it also find typos and security issues in the app. The issue is reported along with severity level thus allowing the developer to fix it based on the priority and level of damage they can cause. It is easy to use and can significantly improve the quality of your code.

Effect of Lint testing on Speed of the Android App

Lint testing can significantly improve the speed of the app in the following ways

  1. Android Link test helps in removing the declaration redundancy in the code, thus the Gradle need not to bind the same object again and again helping to improve the speed.
  2. Lint test helps to find bugs related to Class Structure in different Activities of the Application which is necessary to avoid the case of memory leaks.
  3. Lint testing also tells the developer above the size of resources use for example the drawable resources which sometimes take up a large piece of memory in the application. Cleaning these resources or replacing them with lightweight drawables helps in increasing the speed of the app.
  4. Overall Lint Testing helps in removing Typos, unused import statement, redundant strings, hence refactoring the whole code and increasing stability and speed.

Setup

We can use Gradle to invoke the list test by the following commands in the root directory of the folder.

To set up we can add this code to our build.gradle file

 lintOptions {
        lintConfig file("lint.xml")
    }

The lint.xml generated will look something like this

<?xml version="1.0" encoding="UTF-8"?>
<lint>
   <!-- Changes the severity of these to "error" for getting to a warning-free build -->
   <issue id="UnusedResources" severity="error"/>
</lint>

To explicitly run the test on Android we can use the following commands.

On Windows

gradlew lint

On Mac

./gradlew lint

We can also use the lint testing on various variants of the app, using commands such as

gradle lintDebug 

  or 

gradle lintRelease.

The xml file generated contains the error along with their severity level .

<?xml version="1.0" encoding="UTF-8"?>
<lint>
   <issue id="Invalid Package" severity="ignore" />
   <!-- All below are issues that have been brought to informational (so they are visible, but don't break the build) -->
   <issue id="GradleDependency" severity="informational" />    <issue id="Old TargetApi" severity="informational" />
</lint>

Testing on Susi Android:-

After testing the result on Susi Android we find the following errors.

As we can see that there are two errors and a lot of warnings. Though warning are not that severe but we can definitely improve on this. Thus making a habit of testing your code with lint test will improve the performance of your app and will make it more faster.

The test provides a complete and detailed list of issues present in the project.

We can find the exact location as well as the cause of the error by going deeper into the directory like this.

We can see there is a error in build.gradle file which is due to different versions of libraries present in the gradle files as of com.android.support libraries must be of same version.

In this way we can test out code and find errors in it.

Continue ReadingSetup Lint Testing in SUSI Android

Map Support for SUSI Webchat

SUSI chat client supports map tiles now for queries related to location. SUSI responds with an interactive internal map tile with the location pointed by a marker. It also provides you with a link to open street maps where you can get the whole view of the location using the zooming options provided and also gives the population count for that location.

Lets visit SUSI WebChat and try it out.

Query : Where is london
Response :

Implementation:

How do we know that a map tile is to be rendered?
The actions in the API response tell the client what to render. The client loops through the actions array and renders the response for each action accordingly.

"actions": [
  {
    "type": "answer",
    "expression": "City of London is a place with a population of 7556900.             Here is a map: https://www.openstreetmap.org/#map=13/51.51279067225417/-0.09184009399817228"
  },
  {
    "type": "anchor",
    "link":    "https://www.openstreetmap.org/#map=13/51.51279067225417/-0.09184009399817228",
    "text": "Link to Openstreetmap: City of London"
  },
  {
    "type": "map",
    "latitude": "51.51279067225417",
    "longitude": "-0.09184009399817228",
    "zoom": "13"
  }
]

Note: The API response has been trimmed to show only the relevant content.

The first action element is of type answer so the client renders the text response, ‘City of London is a place with a population of 7556900. Here is a map: https://www.openstreetmap.org/#map=13/51.51279067225417/-0.09184009399817228

The second action element is of type anchor with the text to display and the link to hyperlink specified by the text and link attributes, so the client renders the text `Link to Openstreetmap: City of London`, hyperlinked to “https://www.openstreetmap.org/#map=13/51.51279067225417/-0.09184009399817228”.

Finally, the third action element is of type map. Latitude, Longitude and zoom level information are also  specified using latitude, longitude and zoom attributes. The client renders a map using these attributes.

I used react-leafletmodule to render the interactive map tiles.

To integrate it into our project and set the required style for the map tiles, we need to load Leaflet’s CSS style sheet and we also need to include height and width for the map component. 

<link rel="stylesheet"  href="http://cdn.leafletjs.com/leaflet/v0.7.7/leaflet.css" />
.leaflet-container {
  height: 150px;
  width: 80%;
  margin: 0 auto;
}
case 'map': {

  let lat = parseFloat(data.answers[0].actions[index].latitude);
  let lng = parseFloat(data.answers[0].actions[index].longitude);
  let zoom = parseFloat(data.answers[0].actions[index].zoom);
  let mymap = drawMap(lat,lng,zoom);

  listItems.push(
    <li className='message-list-item' key={action+index}>
      <section className={messageContainerClasses}>
        {mymap}
        <p className='message-time'>
          {message.date.toLocaleTimeString()}
        </p>;
      </section>
    </li>
  );

  break;
}
import { divIcon } from 'leaflet';
import { Map, Marker, Popup, TileLayer } from 'react-leaflet';


// Draw a Map

function drawMap(lat,lng,zoom){

  let position = [lat, lng];

  const icon = divIcon({
    className: 'map-marker-icon',
    iconSize: [35, 35]
    });

  const map = (
    <Map center={position} zoom={zoom}>
      <TileLayer
      attribution=''
      url='http://{s}.tile.osm.org/{z}/{x}/{y}.png'
      />
      <ExtendedMarker position={position} icon={icon}>
        <Popup>
          <span><strong>Hello!</strong> <br/> I am here.</span>
        </Popup>
      </ExtendedMarker>
    </Map>
  );

return map;

}

Here, I used a custom marker icon because the default icon provided by leaflet had an issue and was not being rendered. I used divIcon from leaflet to create a custom map marker icon.

When the map tile is rendered, we see a Popup message at the marker. The extended marker class is used to keep the Popup open initially.

class ExtendedMarker extends Marker {
  componentDidMount() {
    super.componentDidMount();
    this.leafletElement.openPopup();
  }
}


The function drawMap returns a Map tile component which is rendered and we have our interactive map!

Resources
Continue ReadingMap Support for SUSI Webchat

Hyperlinking Support for SUSI Webchat

SUSI responses can contain links or email ids. Whenever we want to access those links or send a mail to those email ids, it is very inconvenient for the user to manually copy the link and check out the contents, which is a very bad UX.

I used a module called ‘react-linkify’ to address this issue.
React-linkify’ is a React component to parse links (urls, emails, etc.) in text into clickable links.

Usage:

<Linkify>{text to linkify}</Linkify>

Any link that appears inside the linkify component will be hyperlinked and is made clickable. It uses regular expressions and pattern matching to detect URLs and mail ids and are made clickable such that clicking on URLs opens the link in a new window and clicking a mail id opens “mailto:” .

Code:

export const parseAndReplace = (text) => {return <Linkify properties={{target:"_blank"}}>{text}</Linkify>;}

Lets visit SUSI WebChat and try it out.

Query: search internet

Response: Internet The global system of interconnected computer networks that use the Internet protocol suite to… https://duckduckgo.com/Internet

The link has been parsed from the response text and has been successfully hyperlinked. Clicking the links opens the respective URL in a new window.

Resources
Continue ReadingHyperlinking Support for SUSI Webchat