Using CoreLocation in SUSI iOS

The SUSI Server responds with intelligent answers to the user’s queries. To make these answers better, the server makes use of the user’s location which is sent as a parameter to the query request each time. To implement this feature in the SUSI iOS client, we use the CoreLocation framework provided by Apple which helps us to get the user’s location coordinates and add them as a parameter to each request made.

In order to start with using the CoreLocation framework, we first import it inside the view controller.

import CoreLocation

Now, we create a variable of type CLLocationManager which will help us to use the actual functionality.

// Location Manager
var locationManager = CLLocationManager()

The location manager has some delegate methods which give an option to get the maximum accuracy for a user’s location.  To set that, we need the controller to conform to the CLLocationManagerDelegate, so we create an extension of the view controller conforming to this.

extension MainViewController: CLLocationManagerDelegate {

   // use functionality

}

Next, we set the manager delegate.

locationManager.delegate = self

And create a method to ask for using the user’s location and set the delegate properties.

func configureLocationManager() {
       locationManager.delegate = self
       if CLLocationManager.authorizationStatus() == .notDetermined || CLLocationManager.authorizationStatus() == .denied {
           self.locationManager.requestWhenInUseAuthorization()
       }

       locationManager.distanceFilter = kCLDistanceFilterNone
       locationManager.desiredAccuracy = kCLLocationAccuracyBest
}

Here, we ask for the user location if it was previously denied or is not yet determined and following that, we set the `distanceFilter` as kCLDistanceFilterNone  and `desiredAccuray` as kCLLocationAccuracyBest.. Finally, we are left with starting to update the location which we do by:

locationManager.startUpdatingLocation()

We call this method inside viewDidLoad to start updation of the location when the view first loads. The complete extension looks like below:

extension MainViewController: CLLocationManagerDelegate {

   // Configures Location Manager
   func configureLocationManager() {
       locationManager.delegate = self
       if CLLocationManager.authorizationStatus() == .notDetermined || CLLocationManager.authorizationStatus() == .denied {
           self.locationManager.requestWhenInUseAuthorization()
       }

       locationManager.distanceFilter = kCLDistanceFilterNone
       locationManager.desiredAccuracy = kCLLocationAccuracyBest
       locationManager.startUpdatingLocation()
   }

}

Now, it’s very easy to use the location manager and get the coordinates and add it to the params for each request.

if let location = locationManager.location {
   params[Client.ChatKeys.Latitude] = location.coordinate.latitude as AnyObject
   params[Client.ChatKeys.Longitude] = location.coordinate.longitude as AnyObject
}

Now the params which is a dictionary object is added to each request made so that the user get’s the most accurate results for each query he makes.

References:

Continue ReadingUsing CoreLocation in SUSI iOS

Calculation of the Frame Size of the Chat Bubble in SUSI iOS

We receive intelligent responses from the SUSI Server based on our query. Each response contains a different set of actions and the content of the action can be of variable sizes, map, string, table, pie chart, etc. To make the chat bubble size dynamic in the SUSI iOS client, we need to check the action type. For each action, we calculate a different frame size which makes the size of the chat bubble dynamic and hence solving the issue of dynamic size of these bubbles.

In order to calculate the frame size, as mentioned above, we need to check the action type of that message. Let’s start by first making the API call sending the query and getting the action types as a response.

func queryResponse(_ params: [String : AnyObject], _ completion: @escaping(_ messages: List<Message>?, _ success: Bool, _ error: String?) -> Void) {

   let url = getApiUrl(UserDefaults.standard.object(forKey: ControllerConstants.UserDefaultsKeys.ipAddress) as! String, Methods.Chat)

       _ = makeRequest(url, .get, [:], parameters: params, completion: { (results, message) in
           if let _ = message {
               completion(nil, false, ResponseMessages.ServerError)
           } else if let results = results {

               guard let response = results as? [String : AnyObject] else {
                   completion(nil, false, ResponseMessages.InvalidParams)
                   return
               }

               let messages = Message.getAllActions(data: response)
               completion(messages, true, nil)
           }
           return
})
}

Here, we are sending the query in the params dictionary. The `makeRequest` method makes the actual API call and returns a results object and an error object if any which default to `nil`. First, we check if the error variable is `nil` or not and if it is, we parse the complete response by using a helper method created in the Message object called `getAllActions`. This basically takes the response and gives us a list of messages of all action types returned for that query.

In order to display this in the UI, we need to call this method in the View Controller to actually use the result. Here is how we call this method.

var params: [String : AnyObject] = [
 Client.WebsearchKeys.Query: inputTextView.text! as AnyObject,
 Client.ChatKeys.TimeZoneOffset: ControllerConstants.timeZone as AnyObject,
       Client.ChatKeys.Language: Locale.current.languageCode as AnyObject
]

if let location = locationManager.location {
 params[Client.ChatKeys.Latitude] = location.coordinate.latitude as AnyObject
       params[Client.ChatKeys.Longitude] = location.coordinate.longitude as AnyObject
}

if let userData = UserDefaults.standard.dictionary(forKey: ControllerConstants.UserDefaultsKeys.user) as [String : AnyObject]? {
 let user = User(dictionary: userData)
        params[Client.ChatKeys.AccessToken] = user.accessToken as AnyObject
       }

Client.sharedInstance.queryResponse(params) { (messages, success, _) in
 DispatchQueue.main.async {
 if success {
 self.collectionView?.performBatchUpdates({
                            for message in messages! {
                                try! self.realm.write {
                                    self.realm.add(message)
                                    self.messages.append(message)
                                    let indexPath = IndexPath(item: self.messages.count - 1, section: 0)
                                    self.collectionView?.insertItems(at: [indexPath])
                                }
                            }
                        }, completion: { (_) in
                            self.scrollToLast()
                     })
 }
 }
}

Here, we are creating a params object sending the query and some additional parameters such as time zone, location coordinates and access token identifying the user. After the response is received, which contains a list of messages, we use a method called `performBatchUpdates` on the collection view where we loop through all the messages, writing each one of them to the database and then adding at the end of the collection view. Here we got all the messages inside the `messages` list object where each message can be checked for its action type and a frame can be calculated for the same.

Since the frame for each cell is returned in the `sizeForItemAt` delegate method of the collectionViewDelegate, we first grab the message using its indexPath and check the action type for each such message added to the collection view.

if message.actionType == ActionType.map.rawValue {
   // map action
} else if message.actionType == ActionType.rss.rawValue {
   // rss action type
} else if message.actionType == ActionType.websearch.rawValue {
   // web search action
}

Since map action will be a static map, we use a hard coded value for the map’s height and the width equals to the width of the cell frame.

CGSize(width: view.frame.width, height: Constants.mapActionHeight)

Next, web search and rss action having the same UI, will have the same frame size but the number of cells inside each of the frame for these action depends on number of responses were received from the server.

CGSize(width: view.frame.width, height: Constants.rssActionHeight)

And the check can be condensed as well, instead of checking each action separately, we use a `||` (pipes) or an `OR`.

else if message.actionType == ActionType.rss.rawValue ||
            message.actionType == ActionType.websearch.rawValue {
           // web search and rss action
 }

The anchor and answer action types, are supposed to display a string in the chat bubble. So the chat bubble size can be calculated using the following method:

let size = CGSize(width: Constants.maxCellWidth, height: Constants.maxCellHeight)
let options = NSStringDrawingOptions.usesFontLeading.union(.usesLineFragmentOrigin)
let estimatedFrame = NSString(string: messageBody).boundingRect(with: size, options: options, attributes: [NSFontAttributeName: UIFont.systemFont(ofSize: Constants.messageFontSize)], context: nil)

Here, we first create a maximum frame size that can exist. Then, using the drawingOptions create an options variable. The actual calculation of the frame happens in the last method where we use the complete message string and this returns us the actual frame for the above action types. Use the above method to get the frame in the `CGRect` type.

Below, is the complete method used for this calculation:

func collectionView(_ collectionView: UICollectionView, layout collectionViewLayout: UICollectionViewLayout, sizeForItemAt indexPath: IndexPath) -> CGSize {
        let message = messages[indexPath.row]
 
        let estimatedFrame = self.estimatedFrame(messageBody: message.message)
        if message.actionType == ActionType.map.rawValue {
            return CGSize(width: view.frame.width, height: Constants.mapActionHeight)
        } else if message.actionType == ActionType.rss.rawValue ||
            message.actionType == ActionType.websearch.rawValue {
            return CGSize(width: view.frame.width, height: Constants.rssActionType)
        }
        return CGSize(width: view.frame.width, height: estimatedFrame.height + Constants.answerActionMargin)
    }

rn CGSize(width: view.frame.width, height: estimatedFrame.height + Constants.answerActionMargin)
}

Below are the results of the app in action.

References:

Continue ReadingCalculation of the Frame Size of the Chat Bubble in SUSI iOS

Implementing Speech To Text in SUSI iOS

SUSI being an intelligent bot has the capabilities by which the user can provide input in a hands-free mode by talking and not requiring to even lift the phone for typing. The speech to text feature is available in SUSI iOS with the help of the Speech framework which was released alongside iOS 10 which enables continuous speech detection and transcription. The detection is really fast and supports around 50 languages and dialects from Arabic to Vietnamese. The speech recognition API does its heavy tasks of detection on Apple’s servers which requires an internet connection. The same API is also not always available on all newer devices and also provides the ability to check if a specific language is supported at a particular time.

How to use the Speech to Text feature?

  • Go to the view controller and import the speech framework
  • Now, because the speech is transmitted over the internet and uses Apple’s servers for computation, we need to ask the user for permissions to use the microphone and speech recognition feature. Add the following two keys to the Info.plist file which displays alerts asking user permission to use speech recognition and for accessing the microphone. Add a specific sentence for each key string which will be displayed to the user in the alerts.
    1. NSSpeechRecognitionUsageDescription
    2. NSMicrophoneUsageDescription

The prompts appear automatically when the functionality is used in the app. Since we already have the Hot word recognition enabled, the microphone alert would show up automatically after login and the speech one shows after the microphone button is tapped.

3) To request the user for authorization for Speech Recognition, we use the method SFSpeechRecognizer.requestAuthorization.

func configureSpeechRecognizer() {
        speechRecognizer?.delegate = self

        SFSpeechRecognizer.requestAuthorization { (authStatus) in
            var isEnabled = false

            switch authStatus {
            case .authorized:
                print("Autorized speech")
                isEnabled = true
            case .denied:
                print("Denied speech")
                isEnabled = false
            case .restricted:
                print("speech restricted")
                isEnabled = false
            case .notDetermined:
                print("not determined")
                isEnabled = false
            }

            OperationQueue.main.addOperation {

                // handle button enable/disable

                self.sendButton.tag = isEnabled ? 0 : 1

                self.addTargetSendButton()
            }
        }
    }

4)   Now, we create instances of the AVAudioEngine, SFSpeechRecognizer, SFSpeechAudioBufferRecognitionRequest,SFSpeechRecognitionTask

let speechRecognizer = SFSpeechRecognizer(locale: Locale.init(identifier: "en-US"))
var recognitionRequest: SFSpeechAudioBufferRecognitionRequest?
var recognitionTask: SFSpeechRecognitionTask?
let audioEngine = AVAudioEngine()

5)  Create a method called `readAndRecognizeSpeech`. Here, we do all the recognition related stuff. We first check if the recognitionTask is running or not and if it does we cancel the task.

if recognitionTask != nil {
  recognitionTask?.cancel()
  recognitionTask = nil
}

6)  Now, create an instance of AVAudioSession to prepare the audio recording where we set the category of the session as recording, the mode and activate it. Since these might throw an exception, they are added inside the do catch block.

let audioSession = AVAudioSession.sharedInstance()

do {

    try audioSession.setCategory(AVAudioSessionCategoryRecord)

    try audioSession.setMode(AVAudioSessionModeMeasurement)

    try audioSession.setActive(true, with: .notifyOthersOnDeactivation)

} catch {

    print("audioSession properties weren't set because of an error.")

}

7)  Instantiate the recognitionRequest.

recognitionRequest = SFSpeechAudioBufferRecognitionRequest()

8) Check if the device has an audio input else throw an error.

guard let inputNode = audioEngine.inputNode else {

fatalError("Audio engine has no input node")

}

9)  Enable recognitionRequest to report partial results and start the recognitionTask.

recognitionRequest.shouldReportPartialResults = true

recognitionTask = speechRecognizer?.recognitionTask(with: recognitionRequest, resultHandler: { (result, error) in

  var isFinal = false // to indicate if final result

  if result != nil {

    self.inputTextView.text = result?.bestTranscription.formattedString

    isFinal = (result?.isFinal)!

  }

  if error != nil || isFinal {

    self.audioEngine.stop()

    inputNode.removeTap(onBus: 0)

    self.recognitionRequest = nil

    self.recognitionTask = nil

  }
})

10) Next, we start with writing the method that performs the actual speech recognition. This will record and process the speech continuously.

  • First, we create a singleton for the incoming audio using .inputNode
  • .installTap configures the node and sets up the buffer size and the format
let recordingFormat = inputNode.outputFormat(forBus: 0)

inputNode.installTap(onBus: 0, bufferSize: 1024, format: recordingFormat) { (buffer, _) in

    self.recognitionRequest?.append(buffer)

}

11)  Next, we prepare and start the audio engine.

audioEngine.prepare()

do {

  try audioEngine.start()

} catch {

  print("audioEngine couldn't start because of an error.")

}

12)  Create a method that stops the Speech recognition.

func stopSTT() {

    print("audioEngine stopped")

    audioEngine.inputNode?.removeTap(onBus: 0)

    audioEngine.stop()

    recognitionRequest?.endAudio()

    indicatorView.removeFromSuperview()



    if inputTextView.text.isEmpty {

        self.sendButton.setImage(UIImage(named: ControllerConstants.mic), for: .normal)

    } else {

        self.sendButton.setImage(UIImage(named: ControllerConstants.send), for: .normal)

    }

        self.inputTextView.isUserInteractionEnabled = true
}

13)  Update the view when the speech recognition is running indicating the user its status. Add below code just below audio engine preparation.

// Listening indicator swift

self.indicatorView.frame = self.sendButton.frame

self.indicatorView.isUserInteractionEnabled = true

let gesture: UITapGestureRecognizer = UITapGestureRecognizer(target: self, action: #selector(startSTT))

gesture.numberOfTapsRequired = 1

self.indicatorView.addGestureRecognizer(gesture)
self.sendButton.setImage(UIImage(), for: .normal)

indicatorView.startAnimating()

self.sendButton.addSubview(indicatorView)

self.sendButton.addConstraintsWithFormat(format: "V:|[v0(24)]|", views: indicatorView)

self.sendButton.addConstraintsWithFormat(format: "H:|[v0(24)]|", views: indicatorView)

self.inputTextView.isUserInteractionEnabled = false

The screenshot of the implementation is below:

       

References

Continue ReadingImplementing Speech To Text in SUSI iOS

Youtube videos in the SUSI iOS Client

The iOS and android client already have the functionality to play videos based on the queries. In order to implement this feature of playing videos in the iOS client, we use the Youtube Data API v3. The task here was to create an UI/UX for the playing of videos within the app. An API call is made initially to fetch the youtube videos based on the query and they video ID of the first object is extracted and used to play the video.

The API endpoint for youtube data API looks like:

https://www.googleapis.com/youtube/v3/search?part=snippet&q={query}&key={your_api_key}

Using this we get the following result: ( I am adding only the first item which is required since the response is too long )

Path: $.items[0]

{

 "kind": "youtube#searchResult",

 "etag": "\"m2yskBQFythfE4irbTIeOgYYfBU/oR-eA572vNoma1XIhrbsFTotfTY\"",

 "id": {

"kind": "youtube#channel",

"channelId": "UCQprMsG-raCIMlBudm20iLQ"

 },

 "snippet": {

"publishedAt": "2015-01-01T11:06:00.000Z",

"channelId": "UCQprMsG-raCIMlBudm20iLQ",

"title": "FOSSASIA",

"description": "FOSSASIA is supporting the development of Free and Open Source technologies for social change in Asia. The annual FOSSASIA Summit brings together ...",

"thumbnails": {

"default": {

"url": "https://yt3.ggpht.com/-CP18cWbo34A/AAAAAAAAAAI/AAAAAAAAAAA/kEmIgO8OjCk/s88-c-k-no-mo-rj-c0xffffff/photo.jpg"

},

"medium": {

"url": "https://yt3.ggpht.com/-CP18cWbo34A/AAAAAAAAAAI/AAAAAAAAAAA/kEmIgO8OjCk/s240-c-k-no-mo-rj-c0xffffff/photo.jpg"

},

"high": {

"url": "https://yt3.ggpht.com/-CP18cWbo34A/AAAAAAAAAAI/AAAAAAAAAAA/kEmIgO8OjCk/s240-c-k-no-mo-rj-c0xffffff/photo.jpg"

}

},

"channelTitle": "FOSSASIA",

"liveBroadcastContent": "upcoming"

 }

}

We parse the above object to grab the videoID, based on the query,  we will use code below:

if let itemsObject = response[Client.YoutubeResponseKeys.Items] as? [[String : AnyObject]] {
    if let items = itemsObject[0][Client.YoutubeResponseKeys.ID] as? [String : AnyObject] {
         let videoID = items[Client.YoutubeResponseKeys.VideoID] as? String
         completion(videoID, true, nil)
    }
}

This videoID is returned to the Controller where this method was called.

Now, we begin with designing the UI for the same. First of all, we need a view on which the youtube video will be played and this view would help dismiss the video by clicking on it.

First, we add the blackView to the entire screen.

// declaration
let blackView = UIView()

// Add backgroundView
func addBackgroundView() {

   If let window = UIApplication.shared.keyWindow {

           self.view.addSubview(blackView) 

           // Cover the entire screen
           blackView.frame = window.frame

           blackView.backgroundColor = UIColor(white: 0, alpha: 0.5)
           blackView.addGestureRecognizer(UITapGestureRecognizer(target: self, action: #selector(handleDismiss)))

   }

}

func handleDismiss() {
   UIView.animate(withDuration: 0.5, delay: 0, usingSpringWithDamping: 1, initialSpringVelocity: 1, options: .curveEaseOut, animations: {
       self.blackView.removeFromSuperview()
   }, completion: nil)
}

Next, we add the YoutubePlayerView. For this we use the Pod `YoutubePlayer`. Since, it had a few warnings showing as well as some videos not being played I had to make fixes to the original pod and use my own customized version ( available here ).

// Youtube Player
lazy var youtubePlayer: YouTubePlayerView = {
    let frame = CGRect(x: 0, y: 0, width: self.view.frame.width - 16, height: self.view.frame.height * 1 / 3)
    let player = YouTubePlayerView(frame: frame)
    return player
}()

// Shows Youtube Player

func addYotubePlayer(_ videoID: String) {
    if let window = UIApplication.shared.keyWindow {

       // Add YoutubePlayer view on top of blackView
        self.blackView.addSubview(self.youtubePlayer)
        // Calculate and set frame

       let centerX = UIScreen.main.bounds.size.width / 2
        let centerY = UIScreen.main.bounds.size.height / 3
        self.youtubePlayer.center = CGPoint(x: centerX, y: centerY)

       // Load Player using the Video ID 
        self.youtubePlayer.loadVideoID(videoID)

        blackView.alpha = 0
        youtubePlayer.alpha = 0

        UIView.animate(withDuration: 0.5, delay: 0, usingSpringWithDamping: 1, initialSpringVelocity: 1, options: .curveEaseOut, animations: {
            self.blackView.alpha = 1
            self.youtubePlayer.alpha = 1
       }, completion: nil)
    }
}

We are set with the UI and the only thing we are left with is to actually call the API in the client and after getting the `videoID` from that we call the above method passing this `videoID`. Before calling we need to check whether our query contains the play action or not and if it does we make the API call and add the player.

if let text = inputTextField.text {
    if text.contains("play") || text.contains("Play") {
        let query = text.replacingOccurrences(of: "play", with: "").replacingOccurrences(of: "Play", with: "")
        Client.sharedInstance.searchYotubeVideos(query) { (videoID, _, _) in
            DispatchQueue.main.async {
                if let videoID = videoID {
                    self.addYotubePlayer(videoID)
                 }
             }
          }
    }

}

We are all set now!Below is the output for the Youtube Player:

Continue ReadingYoutube videos in the SUSI iOS Client

Websearch and Link Preview support in SUSI iOS

The SUSI.AI server responds to API calls with answers to the queries made. These answers might contain an action, for example a web search, where the client needs to make a web search request to fetch different web pages based on the query. Thus, we need to add a link preview in the iOS Client for each such page extracting and displaying the title, description and a main image of the webpage.

At first we make the API call adding the query to the query parameter and get the result from it.

API Call:

http://api.susi.ai/susi/chat.json?timezoneOffset=-330&q=amazon

And get the following result:

{

"query": "amazon",

"count": 1,

"client_id": "aG9zdF8xMDguMTYyLjI0Ni43OQ==",

"query_date": "2017-06-02T14:34:15.675Z",

"answers": [

{

"data": [{

"0": "amazon",

"1": "amazon",

"timezoneOffset": "-330"

}],

"metadata": {

"count": 1

},

"actions": [{

"type": "answer",

"expression": "I don't know how to answer this. Here is a web search result:"

},

{

"type": "websearch",

"query": "amazon"

}]

}],

"answer_date": "2017-06-02T14:34:15.773Z",

"answer_time": 98,

"language": "en",

"session": {

"identity": {

"type": "host",

"name": "108.162.246.79",

"anonymous": true

}

}

}

After parsing this response, we first recognise the type of action that needs to be performed, here we get `websearch` which means we need to make a web search for the query. Here, we use `DuckDuckGo’s` API to get the result.

API Call to DuckDuckGo:

http://api.duckduckgo.com/?q=amazon&format=json

I am adding just the first object of the required data since the API response is too long.

Path: $.RelatedTopics[0]

{

 "Result": "<a href=\"https://duckduckgo.com/Amazon.com\">Amazon.com</a>Amazon.com, also called Amazon, is an American electronic commerce and cloud computing company...",

 "Icon": {

"URL": "https://duckduckgo.com/i/d404ba24.png",

"Height": "",

"Width": ""

},

 "FirstURL": "https://duckduckgo.com/Amazon.com",

 "Text": "Amazon.com Amazon.com, also called Amazon, is an American electronic commerce and cloud computing company..."

}

For the link preview, we need an image logo, URL and a description for the same so, here we will use the `Icon.URL` and `Text` key. We have our own class to parse this data into an object.

class WebsearchResult: NSObject {
   
   var image: String = "no-image"
   var info: String = "No data found"
   var url: String = "https://duckduckgo.com/"
   var query: String = ""
   
   init(dictionary: [String:AnyObject]) {
       
       if let relatedTopics = dictionary[Client.WebsearchKeys.RelatedTopics] as? [[String : AnyObject]] {
           
           if let icon = relatedTopics[0][Client.WebsearchKeys.Icon] as? [String : String] {
               if let image = icon[Client.WebsearchKeys.Icon] {
                   self.image = image
               }
           }
           
           if let url = relatedTopics[0][Client.WebsearchKeys.FirstURL] as? String {
               self.url = url
           }
           
           if let info = relatedTopics[0][Client.WebsearchKeys.Text] as? String {
               self.info = info
           }
           
           if let query = dictionary[Client.WebsearchKeys.Heading] as? String {
               let string = query.lowercased().replacingOccurrences(of: " ", with: "+")
               self.query = string
           }   
       }   
   }   
}

We now have the data and only thing left is to display it in the UI.

Within the Chat Bubble, we need to add a container view which will contain the image, and the text description.

 let websearchContentView = UIView()
    
    let searchImageView: UIImageView = {
        let imageView = UIImageView(frame: CGRect(x: 0, y: 0, width: 44, height: 44))
        imageView.contentMode = .scaleAspectFit

       // Placeholder image assigned
        imageView.image = UIImage(named: "no-image")
        return imageView
    }()
    
    let websiteText: UILabel = {
        let label = UILabel()
        label.textColor = .white
        return label
    }()

   func addLinkPreview(_ frame: CGRect) {
        textBubbleView.addSubview(websearchContentView)
        websearchContentView.backgroundColor = .lightGray
        websearchContentView.frame = frame
        
        websearchContentView.addSubview(searchImageView)
        websearchContentView.addSubview(websiteText)




       // Add constraints in UI
        websearchContentView.addConstraintsWithFormat(format: "H:|-4-[v0(44)]-4-[v1]-4-|", views: searchImageView, websiteText)
        websearchContentView.addConstraintsWithFormat(format: "V:|-4-[v0]-4-|", views: searchImageView)
        websearchContentView.addConstraintsWithFormat(format: "V:|-4-[v0(44)]-4-|", views: websiteText)
    }

Next, in the Collection View, while checking other action types, we add checking for `websearch` and then call the API there followed by adding frame size and calling the `addLinkPreview` function.

else if message.responseType == Message.ResponseTypes.websearch {
  let params = [
    Client.WebsearchKeys.Query: message.query!,
    Client.WebsearchKeys.Format: "json"
  ]

  Client.sharedInstance.websearch(params, { (results, success, error) in                      
    if success {
      cell.message?.websearchData = results
      message.websearchData = results
      self.collectionView?.reloadData()
      self.scrollToLast()
    } else {
      print(error)
    }
  })
                    
  cell.messageTextView.frame = CGRect(x: 16, y: 0, width: estimatedFrame.width + 16, height: estimatedFrame.height + 30)
  

 cell.textBubbleView.frame = CGRect(x: 4, y: -4, width: estimatedFrame.width + 16 + 8 + 16, height: estimatedFrame.height + 20 + 6 + 64)
                    
  let frame = CGRect(x: 16, y: estimatedFrame.height + 20, width: estimatedFrame.width + 16 - 4, height: 60 - 8)
  

 cell.addLinkPreview(frame)

}

And set the collection View cell’s size.

else if message.responseType == Message.ResponseTypes.websearch {
  return CGSize(width: view.frame.width, height: estimatedFrame.height + 20 + 64)
}

And we are done 🙂

Here is the final version how this would look like on the device:

 

Continue ReadingWebsearch and Link Preview support in SUSI iOS

Susi AI Skill Development

What is Susi?

Susi is an open source intelligent personal assistant which has the capability to learn and respond better to queries. It is also capable of making to-do lists, setting alarms, providing weather and traffic info all in real time. Susi responds based on skills.

What is a skill? How do we teach a skill?

A skill is a piece of code which performs a set of actions in order to respond to the user’s query. These skills are based on pattern matching which help them mapping the user’s query to a specific skill and responding accordingly. Teaching a skill to Susi is surprisingly very easy to implement. One can take a look at the Susi Skill Development Tutorial and a video workshop by Michael Christen.

I will try to give a basic idea on how to create a skill, it’s basic structure and some of the skills I developed in the first week.

Prepare to create a skill:

  • Head over to http://dream.susi.ai
  • Create a etherpad with some relevant name
  • Delete all text currently present in there
  • Start writing your skill

Adding to this, for testing a skill one can head over to Susi Web Chat Interface.

Basic Structure for calling an API:

<Regular expression to be matched here>

!console:<response given to the user>
 {
 "url":"<API endpoint>",
 "path":"<Json path here>"
 }
 eol

So, let me explain this line by line.

  1. The regular expression is the one to which the user’s query is matched first.
  2. The console is meant to output the actual response the user sees as response.
  3. In place of the “url”, the API endpoint is passed in.
  4. “path” here specifies how we traverse through the response Json or Jsonp to get the object, starts with “$.”.
  5. At last, “eol” which is the end-of-line marks the end of a skill.

Let’s take an example for better understanding of this:

random gif
!console: $url$
{
    "url" : "http://api.giphy.com/v1/gifs/trending?api_key=dc6zaTOxFJmzC",
    "path" : "$.data[0].images.fixed_height"
}
eol 

 

This skill responds with a link to a random gif.

Steps involved:

  1. Match the string “random gif” with the user’s query.
  2. On successful match, make an API call to the API endpoint specified in “url”
  3. On response, extract the object at the specified path in the json under “path”
  4. Respond to the user with the “url” key’s value which would here be an URL of a GIF.

Let’s try it out on Susi Web Chat. For this, you will first have to load your skill using the dream command followed by etherpad name: dream <etherpad name>. And then you can start testing your skill.

So, we queried “random gif” and we got a response “Click Here!”. The complete URL didn’t show up because all the URLs are currently parsed and a hyperlink for each is created. So try clicking on it to find a GIF.

 

Now, let’s look at one more skill I developed during this period.

# Returns the name of the president of a country

 president of *|who is the president of *| president *
 !console:$plaintext$
 {      "url":"https://api.wolframalpha.com/v2/query?input=president+$1$&output=JSON&appid=9WA6XR-26EWTGEVTE&includepodid=Result",
   "path" : "$.queryresult.pods[0].subpods[0]"
 }
 eol

 

Let’s understand this step by step:

  1. We have here “president of *|who is the president of *| president *”, which means the user’s query matches with anyone of the following because of the use of pipe symbol “|”. The “*” here replaces a word or a list of words, which can be accessed like: “${index}$”  where index is replaced by the position of the “*” in the expression starting from 1.
  2. Now we have something new in the URL. See that  $1$  inside the URL? On runtime, that is replaced with the content of the “*” variable. So if a user puts in query like: “president of usa”, “usa” is mapped to $1$ and is replaced in the URL and appropriate API request is made.
  3. Then the path is traversed in the json response and the value of the “plaintext” key is used to respond to the user.

 

It’s now time to try it out on Susi Web Chat.

So, we got our desired response here, i.e., the name of the president of usa.

Continue ReadingSusi AI Skill Development