Author: Chashmeet Singh

This author has written 16 articles
Read more about the article Using CoreLocation in SUSI iOS

Using CoreLocation in SUSI iOS

The SUSI Server responds with intelligent answers to the user’s queries. To make these answers better, the server makes use of the user’s location which is sent as a parameter to the query request each time. To implement this feature in the SUSI iOS client, we use the CoreLocation framework provided by Apple which helps us to get the user’s location coordinates and add them as a parameter to each request made. In order to start with using the CoreLocation framework, we first import it inside the view controller. import CoreLocation Now, we create a variable of type CLLocationManager which will help us to use the actual functionality. // Location Manager var locationManager = CLLocationManager() The location manager has some delegate methods which give an option to get the maximum accuracy for a user’s location.  To set that, we need the controller to conform to the CLLocationManagerDelegate, so we create an extension of the view controller conforming to this. extension MainViewController: CLLocationManagerDelegate {    // use functionality } Next, we set the manager delegate. locationManager.delegate = self And create a method to ask for using the user’s location and set the delegate properties. func configureLocationManager() {        locationManager.delegate = self        if CLLocationManager.authorizationStatus() == .notDetermined || CLLocationManager.authorizationStatus() == .denied {            self.locationManager.requestWhenInUseAuthorization()        }        locationManager.distanceFilter = kCLDistanceFilterNone        locationManager.desiredAccuracy = kCLLocationAccuracyBest } Here, we ask for the user location if it was previously denied or is not yet determined and following that, we set the `distanceFilter` as kCLDistanceFilterNone  and `desiredAccuray` as kCLLocationAccuracyBest.. Finally, we are left with starting to update the location which we do by: locationManager.startUpdatingLocation() We call this method inside viewDidLoad to start updation of the location when the view first loads. The complete extension looks like below: extension MainViewController: CLLocationManagerDelegate {    // Configures Location Manager    func configureLocationManager() {        locationManager.delegate = self        if CLLocationManager.authorizationStatus() == .notDetermined || CLLocationManager.authorizationStatus() == .denied {            self.locationManager.requestWhenInUseAuthorization()        }        locationManager.distanceFilter = kCLDistanceFilterNone        locationManager.desiredAccuracy = kCLLocationAccuracyBest        locationManager.startUpdatingLocation()    } } Now, it’s very easy to use the location manager and get the coordinates and add it to the params for each request. if let location = locationManager.location {    params[Client.ChatKeys.Latitude] = location.coordinate.latitude as AnyObject    params[Client.ChatKeys.Longitude] = location.coordinate.longitude as AnyObject } Now the params which is a dictionary object is added to each request made so that the user get’s the most accurate results for each query he makes. References: https://developer.apple.com/documentation/corelocation/cllocationmanager https://developer.apple.com/documentation/corelocation/cllocationmanagerdelegate https://www.raywenderlich.com/136165/core-location-geofencing-tutorial

Continue ReadingUsing CoreLocation in SUSI iOS
Read more about the article Calculation of the Frame Size of the Chat Bubble in SUSI iOS

Calculation of the Frame Size of the Chat Bubble in SUSI iOS

We receive intelligent responses from the SUSI Server based on our query. Each response contains a different set of actions and the content of the action can be of variable sizes, map, string, table, pie chart, etc. To make the chat bubble size dynamic in the SUSI iOS client, we need to check the action type. For each action, we calculate a different frame size which makes the size of the chat bubble dynamic and hence solving the issue of dynamic size of these bubbles. In order to calculate the frame size, as mentioned above, we need to check the action type of that message. Let’s start by first making the API call sending the query and getting the action types as a response. func queryResponse(_ params: [String : AnyObject], _ completion: @escaping(_ messages: List<Message>?, _ success: Bool, _ error: String?) -> Void) {    let url = getApiUrl(UserDefaults.standard.object(forKey: ControllerConstants.UserDefaultsKeys.ipAddress) as! String, Methods.Chat)        _ = makeRequest(url, .get, [:], parameters: params, completion: { (results, message) in            if let _ = message {                completion(nil, false, ResponseMessages.ServerError)            } else if let results = results {                guard let response = results as? [String : AnyObject] else {                    completion(nil, false, ResponseMessages.InvalidParams)                    return                }                let messages = Message.getAllActions(data: response)                completion(messages, true, nil)            }            return }) } Here, we are sending the query in the params dictionary. The `makeRequest` method makes the actual API call and returns a results object and an error object if any which default to `nil`. First, we check if the error variable is `nil` or not and if it is, we parse the complete response by using a helper method created in the Message object called `getAllActions`. This basically takes the response and gives us a list of messages of all action types returned for that query. In order to display this in the UI, we need to call this method in the View Controller to actually use the result. Here is how we call this method. var params: [String : AnyObject] = [ Client.WebsearchKeys.Query: inputTextView.text! as AnyObject, Client.ChatKeys.TimeZoneOffset: ControllerConstants.timeZone as AnyObject,       Client.ChatKeys.Language: Locale.current.languageCode as AnyObject ] if let location = locationManager.location { params[Client.ChatKeys.Latitude] = location.coordinate.latitude as AnyObject       params[Client.ChatKeys.Longitude] = location.coordinate.longitude as AnyObject } if let userData = UserDefaults.standard.dictionary(forKey: ControllerConstants.UserDefaultsKeys.user) as [String : AnyObject]? { let user = User(dictionary: userData)        params[Client.ChatKeys.AccessToken] = user.accessToken as AnyObject       } Client.sharedInstance.queryResponse(params) { (messages, success, _) in DispatchQueue.main.async { if success { self.collectionView?.performBatchUpdates({                            for message in messages! {                                try! self.realm.write {                                    self.realm.add(message)                                    self.messages.append(message)                                    let indexPath = IndexPath(item: self.messages.count - 1, section: 0)                                    self.collectionView?.insertItems(at: [indexPath])                                }                            }                        }, completion: { (_) in                            self.scrollToLast()                     }) } } } Here, we are creating a params object sending the query and some additional parameters such as time zone, location coordinates and access token identifying the user. After the response is received, which contains a list of messages, we use a method called `performBatchUpdates` on the collection view where we loop through all the messages, writing each one of them to the database and then adding at the end of the collection…

Continue ReadingCalculation of the Frame Size of the Chat Bubble in SUSI iOS
Read more about the article Implementing Speech To Text in SUSI iOS

Implementing Speech To Text in SUSI iOS

SUSI being an intelligent bot has the capabilities by which the user can provide input in a hands-free mode by talking and not requiring to even lift the phone for typing. The speech to text feature is available in SUSI iOS with the help of the Speech framework which was released alongside iOS 10 which enables continuous speech detection and transcription. The detection is really fast and supports around 50 languages and dialects from Arabic to Vietnamese. The speech recognition API does its heavy tasks of detection on Apple’s servers which requires an internet connection. The same API is also not always available on all newer devices and also provides the ability to check if a specific language is supported at a particular time. How to use the Speech to Text feature? Go to the view controller and import the speech framework Now, because the speech is transmitted over the internet and uses Apple’s servers for computation, we need to ask the user for permissions to use the microphone and speech recognition feature. Add the following two keys to the Info.plist file which displays alerts asking user permission to use speech recognition and for accessing the microphone. Add a specific sentence for each key string which will be displayed to the user in the alerts. NSSpeechRecognitionUsageDescription NSMicrophoneUsageDescription The prompts appear automatically when the functionality is used in the app. Since we already have the Hot word recognition enabled, the microphone alert would show up automatically after login and the speech one shows after the microphone button is tapped. 3) To request the user for authorization for Speech Recognition, we use the method SFSpeechRecognizer.requestAuthorization. func configureSpeechRecognizer() {         speechRecognizer?.delegate = self         SFSpeechRecognizer.requestAuthorization { (authStatus) in             var isEnabled = false             switch authStatus {             case .authorized:                 print("Autorized speech")                 isEnabled = true             case .denied:                 print("Denied speech")                 isEnabled = false             case .restricted:                 print("speech restricted")                 isEnabled = false             case .notDetermined:                 print("not determined")                 isEnabled = false             }             OperationQueue.main.addOperation {                 // handle button enable/disable                 self.sendButton.tag = isEnabled ? 0 : 1                 self.addTargetSendButton()             }         }     } 4)   Now, we create instances of the AVAudioEngine, SFSpeechRecognizer, SFSpeechAudioBufferRecognitionRequest,SFSpeechRecognitionTask let speechRecognizer = SFSpeechRecognizer(locale: Locale.init(identifier: "en-US")) var recognitionRequest: SFSpeechAudioBufferRecognitionRequest? var recognitionTask: SFSpeechRecognitionTask? let audioEngine = AVAudioEngine() 5)  Create a method called `readAndRecognizeSpeech`. Here, we do all the recognition related stuff. We first check if the recognitionTask is running or not and if it does we cancel the task. if recognitionTask != nil { recognitionTask?.cancel()   recognitionTask = nil } 6)  Now, create an instance of AVAudioSession to prepare the audio recording where we set the category of the session as recording, the mode and activate it. Since these might throw an exception, they are added inside the do catch block. let audioSession = AVAudioSession.sharedInstance() do { try audioSession.setCategory(AVAudioSessionCategoryRecord) try audioSession.setMode(AVAudioSessionModeMeasurement) try audioSession.setActive(true, with: .notifyOthersOnDeactivation) } catch { print("audioSession properties weren't set because of an error.") } 7)  Instantiate the recognitionRequest. recognitionRequest…

Continue ReadingImplementing Speech To Text in SUSI iOS

Youtube videos in the SUSI iOS Client

The iOS and android client already have the functionality to play videos based on the queries. In order to implement this feature of playing videos in the iOS client, we use the Youtube Data API v3. The task here was to create an UI/UX for the playing of videos within the app. An API call is made initially to fetch the youtube videos based on the query and they video ID of the first object is extracted and used to play the video. The API endpoint for youtube data API looks like: https://www.googleapis.com/youtube/v3/search?part=snippet&q={query}&key={your_api_key} Using this we get the following result: ( I am adding only the first item which is required since the response is too long ) Path: $.items[0] {  "kind": "youtube#searchResult",  "etag": "\"m2yskBQFythfE4irbTIeOgYYfBU/oR-eA572vNoma1XIhrbsFTotfTY\"",  "id": { "kind": "youtube#channel", "channelId": "UCQprMsG-raCIMlBudm20iLQ"  },  "snippet": { "publishedAt": "2015-01-01T11:06:00.000Z", "channelId": "UCQprMsG-raCIMlBudm20iLQ", "title": "FOSSASIA", "description": "FOSSASIA is supporting the development of Free and Open Source technologies for social change in Asia. The annual FOSSASIA Summit brings together ...", "thumbnails": { "default": { "url": "https://yt3.ggpht.com/-CP18cWbo34A/AAAAAAAAAAI/AAAAAAAAAAA/kEmIgO8OjCk/s88-c-k-no-mo-rj-c0xffffff/photo.jpg" }, "medium": { "url": "https://yt3.ggpht.com/-CP18cWbo34A/AAAAAAAAAAI/AAAAAAAAAAA/kEmIgO8OjCk/s240-c-k-no-mo-rj-c0xffffff/photo.jpg" }, "high": { "url": "https://yt3.ggpht.com/-CP18cWbo34A/AAAAAAAAAAI/AAAAAAAAAAA/kEmIgO8OjCk/s240-c-k-no-mo-rj-c0xffffff/photo.jpg" } }, "channelTitle": "FOSSASIA", "liveBroadcastContent": "upcoming"  } } We parse the above object to grab the videoID, based on the query,  we will use code below: if let itemsObject = response[Client.YoutubeResponseKeys.Items] as? [[String : AnyObject]] {    if let items = itemsObject[0][Client.YoutubeResponseKeys.ID] as? [String : AnyObject] {         let videoID = items[Client.YoutubeResponseKeys.VideoID] as? String         completion(videoID, true, nil)    } } This videoID is returned to the Controller where this method was called. Now, we begin with designing the UI for the same. First of all, we need a view on which the youtube video will be played and this view would help dismiss the video by clicking on it. First, we add the blackView to the entire screen. // declaration let blackView = UIView() // Add backgroundView func addBackgroundView() {    If let window = UIApplication.shared.keyWindow {            self.view.addSubview(blackView)             // Cover the entire screen            blackView.frame = window.frame            blackView.backgroundColor = UIColor(white: 0, alpha: 0.5)            blackView.addGestureRecognizer(UITapGestureRecognizer(target: self, action: #selector(handleDismiss)))    } } func handleDismiss() {    UIView.animate(withDuration: 0.5, delay: 0, usingSpringWithDamping: 1, initialSpringVelocity: 1, options: .curveEaseOut, animations: {        self.blackView.removeFromSuperview()    }, completion: nil) } Next, we add the YoutubePlayerView. For this we use the Pod `YoutubePlayer`. Since, it had a few warnings showing as well as some videos not being played I had to make fixes to the original pod and use my own customized version ( available here ). // Youtube Player lazy var youtubePlayer: YouTubePlayerView = {    let frame = CGRect(x: 0, y: 0, width: self.view.frame.width - 16, height: self.view.frame.height * 1 / 3)    let player = YouTubePlayerView(frame: frame)    return player }() // Shows Youtube Player func addYotubePlayer(_ videoID: String) {    if let window = UIApplication.shared.keyWindow {        // Add YoutubePlayer view on top of blackView        self.blackView.addSubview(self.youtubePlayer)        // Calculate and set frame        let centerX = UIScreen.main.bounds.size.width / 2        let centerY = UIScreen.main.bounds.size.height / 3        self.youtubePlayer.center = CGPoint(x: centerX, y: centerY)        // Load Player using the Video ID        self.youtubePlayer.loadVideoID(videoID)        blackView.alpha = 0        youtubePlayer.alpha = 0        UIView.animate(withDuration: 0.5, delay: 0, usingSpringWithDamping:…

Continue ReadingYoutube videos in the SUSI iOS Client

Websearch and Link Preview support in SUSI iOS

The SUSI.AI server responds to API calls with answers to the queries made. These answers might contain an action, for example a web search, where the client needs to make a web search request to fetch different web pages based on the query. Thus, we need to add a link preview in the iOS Client for each such page extracting and displaying the title, description and a main image of the webpage. At first we make the API call adding the query to the query parameter and get the result from it. API Call: http://api.susi.ai/susi/chat.json?timezoneOffset=-330&q=amazon And get the following result: { "query": "amazon", "count": 1, "client_id": "aG9zdF8xMDguMTYyLjI0Ni43OQ==", "query_date": "2017-06-02T14:34:15.675Z", "answers": [ { "data": [{ "0": "amazon", "1": "amazon", "timezoneOffset": "-330" }], "metadata": { "count": 1 }, "actions": [{ "type": "answer", "expression": "I don't know how to answer this. Here is a web search result:" }, { "type": "websearch", "query": "amazon" }] }], "answer_date": "2017-06-02T14:34:15.773Z", "answer_time": 98, "language": "en", "session": { "identity": { "type": "host", "name": "108.162.246.79", "anonymous": true } } } After parsing this response, we first recognise the type of action that needs to be performed, here we get `websearch` which means we need to make a web search for the query. Here, we use `DuckDuckGo’s` API to get the result. API Call to DuckDuckGo: http://api.duckduckgo.com/?q=amazon&format=json I am adding just the first object of the required data since the API response is too long. Path: $.RelatedTopics[0] {  "Result": "<a href=\"https://duckduckgo.com/Amazon.com\">Amazon.com</a>Amazon.com, also called Amazon, is an American electronic commerce and cloud computing company...",  "Icon": { "URL": "https://duckduckgo.com/i/d404ba24.png", "Height": "", "Width": "" },  "FirstURL": "https://duckduckgo.com/Amazon.com",  "Text": "Amazon.com Amazon.com, also called Amazon, is an American electronic commerce and cloud computing company..." } For the link preview, we need an image logo, URL and a description for the same so, here we will use the `Icon.URL` and `Text` key. We have our own class to parse this data into an object. class WebsearchResult: NSObject {        var image: String = "no-image"    var info: String = "No data found"    var url: String = "https://duckduckgo.com/"    var query: String = ""        init(dictionary: [String:AnyObject]) {                if let relatedTopics = dictionary[Client.WebsearchKeys.RelatedTopics] as? [[String : AnyObject]] {                        if let icon = relatedTopics[0][Client.WebsearchKeys.Icon] as? [String : String] {                if let image = icon[Client.WebsearchKeys.Icon] {                    self.image = image                }            }                        if let url = relatedTopics[0][Client.WebsearchKeys.FirstURL] as? String {                self.url = url            }                        if let info = relatedTopics[0][Client.WebsearchKeys.Text] as? String {                self.info = info            }                        if let query = dictionary[Client.WebsearchKeys.Heading] as? String {                let string = query.lowercased().replacingOccurrences(of: " ", with: "+")                self.query = string            }           }       }    } We now have the data and only thing left is to display it in the UI. Within the Chat Bubble, we need to add a container view which will contain the image, and the text description.  let websearchContentView = UIView()        let searchImageView: UIImageView = {        let imageView = UIImageView(frame: CGRect(x: 0, y: 0, width: 44, height: 44))        imageView.contentMode = .scaleAspectFit        // Placeholder image assigned…

Continue ReadingWebsearch and Link Preview support in SUSI iOS

Susi AI Skill Development

What is Susi? Susi is an open source intelligent personal assistant which has the capability to learn and respond better to queries. It is also capable of making to-do lists, setting alarms, providing weather and traffic info all in real time. Susi responds based on skills. What is a skill? How do we teach a skill? A skill is a piece of code which performs a set of actions in order to respond to the user’s query. These skills are based on pattern matching which help them mapping the user’s query to a specific skill and responding accordingly. Teaching a skill to Susi is surprisingly very easy to implement. One can take a look at the Susi Skill Development Tutorial and a video workshop by Michael Christen. I will try to give a basic idea on how to create a skill, it’s basic structure and some of the skills I developed in the first week. Prepare to create a skill: Head over to http://dream.susi.ai Create a etherpad with some relevant name Delete all text currently present in there Start writing your skill Adding to this, for testing a skill one can head over to Susi Web Chat Interface. Basic Structure for calling an API: <Regular expression to be matched here> !console:<response given to the user> { "url":"<API endpoint>", "path":"<Json path here>" } eol So, let me explain this line by line. The regular expression is the one to which the user’s query is matched first. The console is meant to output the actual response the user sees as response. In place of the “url”, the API endpoint is passed in. “path” here specifies how we traverse through the response Json or Jsonp to get the object, starts with “$.”. At last, “eol” which is the end-of-line marks the end of a skill. Let’s take an example for better understanding of this: random gif !console: $url$ {    "url" : "http://api.giphy.com/v1/gifs/trending?api_key=dc6zaTOxFJmzC",    "path" : "$.data[0].images.fixed_height" } eol   This skill responds with a link to a random gif. Steps involved: Match the string “random gif” with the user’s query. On successful match, make an API call to the API endpoint specified in “url” On response, extract the object at the specified path in the json under “path” Respond to the user with the “url” key’s value which would here be an URL of a GIF. Let’s try it out on Susi Web Chat. For this, you will first have to load your skill using the dream command followed by etherpad name: dream <etherpad name>. And then you can start testing your skill. So, we queried “random gif” and we got a response “Click Here!”. The complete URL didn’t show up because all the URLs are currently parsed and a hyperlink for each is created. So try clicking on it to find a GIF.   Now, let’s look at one more skill I developed during this period. # Returns the name of the president of a country president of *|who is the president of…

Continue ReadingSusi AI Skill Development