Added “table” type action support in SUSI android app

SUSI.AI has many actions supported by it, for eg: answer, anchor, map, piechart, websearch and rss.These actions are a few of those that can be supported in the SUSI.AI android app, but there are many actions implemented on the server side and the web client even has the implementation of how to handle the “table” type response.

The table response is generally a JSON array response with different json objects, where each json object have similar keys, and the actions key in the JSON response has the columns of the table response which are nothing but the keys in the data object of the response.

To implement the table type response in the susi android app a separate file needed to made to parse the table type response, since the keys and values both are required to the display the response. The file ParseTableSusiResponseHelper.kt was made which parsed the JSON object using the Gson converter factory to get the key value of the actions :

“actions”: [

       {

         “columns”: {

           “ingredients”: “Ingredients”,

           “href”: “Instructions Link”,

           “title”: “Recipe”

         },

         “count”: -1,

         “type”: “table”

       }

     ]

 

The inside the columns the keys and the values, both were extracted, values were to displayed in the title of the column and keys used were to extract the values from the “data” object of the response.

The files TableColumn.java, TableData.java are POJO classes that were used for storing the table columns and the data respectively. The TableDatas.java class was used to store the column list and the data list for the table response.

To fetch the table type response from the server a TableSusiResponse.kt file was added that contained serializable entities which were used to map the response values fetched from the server. A variable that contained the data stored in the “answers” key of the response was made of type of an ArrayList of TableAnswers.

@SerializedName(“answers”)
@Expose
val answers: List<TableAnswer> = ArrayList()

The TableAnswer.kt is another file added that contains serializable variables to store values inside the keys of the “answers” object. The actions object shown above is inside the answers object and it was stored in the form of an ArrayList of TableAction.

@SerializedName(“actions”)
@Expose
val actions: List<TableAction> = ArrayList()

Similar to TableAnswer.kt file TableAction.kt file also contains serializable variables that map the values stored in the “actions” object.

In the retrofit service interface SusiService.java a new call was added to fetch the data from the server as follows :

@GET(“/susi/chat.json”)
Call<TableSusiResponse> getTableSusiResponse(@Query(“timezoneOffset”) int timezoneOffset,
                                           @Query(“longitude”) double longitude,
                                           @Query(“latitude”) double latitude,
                                           @Query(“geosource”) String geosource,
                                           @Query(“language”) String language,
                                           @Query(“q”) String query);

Now, after the data was fetched, the table response can be parsed using the Gson converter factory in the ParseTableSusiResponseHelper.kt file. Below is the implementation :

fun parseSusiResponse(response: Response<TableSusiResponse>) {
  try {
      var response1 = Gson().toJson(response)
      var tableresponse = Gson().fromJson(response1, TableBody::class.java)
      for (tableanswer in tableresponse.body.answers) {
          for (answer in tableanswer.actions) {
              var map = answer.columns
              val set = map?.entries
              val iterator = set?.iterator()
              while (iterator?.hasNext().toString().toBoolean()) {
                  val entry = iterator?.next()
                  listColumn.add(entry?.key.toString())
                  listColVal.add(entry?.value.toString())
              }
          }
          val map2 = tableanswer.data
          val iterator2 = map2?.iterator()
          while (iterator2?.hasNext().toString().toBoolean()) {
              val entry2 = iterator2?.next()
              count++;
              for (count in 0..listColumn.size – 1) {
                  val obj = listColumn.get(count)
                  listTableData.add(entry2?.get(obj).toString())
              }
          }
          tableData = TableDatas(listColVal, listTableData)
      }
  } catch (e: Exception) {
      tableData = null
  }
}

 

Now the data is also parsed, we pass the two lists the ColumnList and DataList to the variable of TableDatas.

Three viewholder classes were added to display the table response properly in the app and corresponding to these viewholders a couple of adapters were also made that are responsible for setting the values in the recyclerview present in the views. The first viewholder is the TableViewHolder, it contains the horizontal recyclerview that is used to display the items fetched from the “data” object of the response. The recyclerview in the TableViewHolder has each entity of the type TabViewHolder, this is a simple cardview but also contains another recyclerview inside it which is used to store the keys and values of each of the object inside the “data” object.

TableViewHolder.java file has a setView() method that uses the the object of ChatMessage to get the list of columns and data to be set in the view.

 

Changes were made in the ChatPresenter.kt file to catch the tableresponse when a table type action is detected. Below is the implementation :

if (response.body().answers[0].actions[i].type.equals(“table”)) {
  tableResponse(query)
  return
}

The tableResponse function is as follows :

fun tableResponse(query: String) {
  val tz = TimeZone.getDefault()
  val now = Date()
  val timezoneOffset = -1 * (tz.getOffset(now.time) / 60000)
  val language = if (PrefManager.getString(Constant.LANGUAGE, Constant.DEFAULT).equals(Constant.DEFAULT)) Locale.getDefault().language else PrefManager.getString(Constant.LANGUAGE, Constant.DEFAULT)
  chatModel.getTableSusiMessage(timezoneOffset, longitude, latitude, source, language, query, this)
}

 

It calls the chatModel to get the list of columns and data to be set. The ChatFeedRecyclerAdapter.java files checks for the table response code, and if it matches then the view used for displaying SUSI’s message is the TableViewHolder. Here is how this viewholder is inflated :

case TABLE:
  view = inflater.inflate(R.layout.susi_table, viewGroup, false);
  return new TableViewHolder(view, clickListener);

Below is the final result when the table response is fetched for the query “Bayern munich team players” is :

References :

  1. SUSI server response for table query : https://api.susi.ai/susi/chat.json?timezoneOffset=-330&q=barcelona+team+players
  2. GSON for converting java objects to JSON and JSON to java : http://www.vogella.com/tutorials/JavaLibrary-Gson/article.html
Continue Reading Added “table” type action support in SUSI android app

Adding Support for Displaying Images in SUSI iOS

SUSI provided exciting features in the chat screen. The user can ask a different type of queries and SUSI respond according to the user query. If the user wants to get news, the user can just ask news and SUSI respond with the news. Like news, SUSI can respond to so many queries. Even user can ask SUSI for the image of anything. In response, SUSI displays the image that the user asked. Exciting? Let’s see how displaying images in the chat screen of SUSI iOS is implemented.

Getting image URL from server side –

When we ask susi for the image of anything, it returns the URL of image source in response with answer action type. We get the URL of the image in the expression key of actions object as below:

actions:
[
{
language: "en",
type: "answer",
expression:
"https://pixabay.com/get/e835b60f2cf6033ed1584d05fb1d4790e076e7d610ac104496f1c279a0e8b0ba_640.jpg"
}
]

 

Implementation in App –

In the app, we check if the response coming from server is image URL or not by following two methods.

One – Check if the expression is a valid URL:

func isValidURL() -> Bool {
if let url = URL(string: self) {
return UIApplication.shared.canOpenURL(url)
}
return false
}

The method above return boolean value if the expression is valid or not.

Two – Check if the expression contains image source in URL:

func isImage() -> Bool {
let imageFormats = ["jpg", "jpeg", "png", "gif"]

if let ext = self.getExtension() {
return imageFormats.contains(ext)
}

return false
}

The method above returns the boolean value if the URL string is image source of not by checking its extension.

If both the methods return true, the expression is a valid URL and contain image source, then we consider it as an Image and proceed further.

In collectionView of the chat screen, we register ImageCell and present the cell if the response is the image as below.

Registering the Cell –

register(_:forCellWithReuseIdentifier:) method registers a class for use in creating new collection view cells.

collectionView?.register(ImageCell.self, forCellWithReuseIdentifier: ControllerConstants.imageCell)

Presenting the Cell using cellForItemAt method of UICollectionView

The implementation of cellForItemAt method is responsible for creating, configuring, and returning the appropriate cell for the given item. We do this by calling the dequeueReusableCell(withReuseIdentifier:for:) method of the collection view and passing the reuse identifier that corresponds to the cell type we want. That method always returns a valid cell object. Upon receiving the cell, we set any properties that correspond to the data of the corresponding item, perform any additional needed configuration, and return the cell.

if let expression = message.answerData?.expression, expression.isValidURL(), expression.isImage() {
if let cell = collectionView.dequeueReusableCell(withReuseIdentifier: ControllerConstants.imageCell, for: indexPath) as? ImageCell {
cell.message = message
return cell
}
}

In ImageCell, we present a UIImageView that display the image. When cell the message var, it call downloadImage method to catch and display image from server URL using Kingfisher method.

In method below –

  1. Get image URL string and check if it is image URL
  2. Convert image string to image URL
  3. Set image to the imageView
func downloadImage() {
if let imageString = message?.answerData?.expression, imageString.isImage() {
if let url = URL(string: imageString) {
imageView.kf.setImage(with: url, placeholder: ControllerConstants.Images.placeholder, options: nil, progressBlock: nil, completionHandler: nil)
}
}
}

We have added a tap gesture to the imageView so that when the user taps the image, it opens the full image in the browser by using the method below:

@objc func openImage() {
if let imageURL = message?.answerData?.expression {
if let url = URL(string: imageURL) {
UIApplication.shared.open(url, options: [:], completionHandler: nil)
}
}
}

 

Final Output –

Resources –

  1. Apple’s Documentations on collectionView(_:cellForItemAt:) method
  2. Apple’s Documentations on register(_:forCellWithReuseIdentifier:) method
  3. SUSI iOS Link: https://github.com/fossasia/susi_iOS
  4. SUSI API Link: http://api.susi.ai/
  5. Kingfisher – A lightweight, pure-Swift library for downloading and caching images from the web
Continue Reading Adding Support for Displaying Images in SUSI iOS

Save Chat Messages using Realm in SUSI iOS

Fetching data from the server each time causes a network load which makes the app depend on the server and the network in order to display data. We use an offline database to store chat messages so that we can show messages to the user even if network is not present which makes the user experience better. Realm is used as a data storage solution due to its ease of usability and also, since it’s faster and more efficient to use. So in order to save messages received from the server locally in a database in SUSI iOS, we are using Realm and the reasons for using the same are mentioned below.

The major upsides of Realm are:

  • It’s absolutely free of charge,
  • Fast, and easy to use.
  • Unlimited use.
  • Work on its own persistence engine for speed and performance

Below are the steps to install and use Realm in the iOS Client:

Installation:

  • Install Cocoapods
  • Run `pod repo update` in the root folder
  • In your Podfile, add use_frameworks! and pod ‘RealmSwift’ to your main and test targets.
  • From the command line run `pod install`
  • Use the `.xcworkspace` file generated by Cocoapods in the project folder alongside `.xcodeproj` file

After installation we start by importing `Realm` in the `AppDelegate` file and start configuring Realm as below:

func initializeRealm() {
        var config = Realm.Configuration(schemaVersion: 1,
            migrationBlock: { _, oldSchemaVersion in
                if (oldSchemaVersion < 0) {
                    // Nothing to do!
                }
        })
        config.fileURL = config.fileURL?.deletingLastPathComponent().appendingPathComponent("susi.realm")
        Realm.Configuration.defaultConfiguration = config
}

Next, let’s head over to creating a few models which will be used to save the data to the DB as well as help retrieving that data so that it can be easily used. Since Susi server has a number of action types, we will cover some of the action types, their model and how they are used to store and retrieve data. Below are the currently available data types, that the server supports.

enum ActionType: String {
  case answer
  case websearch
  case rss
  case table
  case map 
  case anchor
}

Let’s start with the creation of the base model called `Message`. To make it a RealmObject, we import `RealmSwift` and inherit from `Object`

class Message: Object {
  dynamic var queryDate = NSDate()
  dynamic var answerDate = NSDate()
  dynamic var message: String = ""
  dynamic var fromUser = true
  dynamic var actionType = ActionType.answer.rawValue
  dynamic var answerData: AnswerAction?
  dynamic var mapData: MapAction?
  dynamic var anchorData: AnchorAction?
}

Let’s study these properties of the message one by one.

  • `queryDate`: saves the date-time the query was made
  • `answerDate`: saves the date-time the query response was received
  • `message`: stores the query/message that was sent to the server
  • `fromUser`: a boolean which keeps track who created the message
  • `actionType`: stores the action type
  • `answerData`, `rssData`, `mapData`, `anchorData` are the data objects that actually store the respective action’s data

To initialize this object, we need to create a method that takes input the data received from the server.

// saves query and answer date
if let queryDate = data[Client.ChatKeys.QueryDate] as? String,
let answerDate = data[Client.ChatKeys.AnswerDate] as? String {
  message.queryDate = dateFormatter.date(from: queryDate)! as NSDate
  message.answerDate = dateFormatter.date(from: answerDate)! as NSDate}if let type = action[Client.ChatKeys.ResponseType] as? String,
  let data = answers[0][Client.ChatKeys.Data] as? [[String : AnyObject]] {
  if type == ActionType.answer.rawValue {
     message.message = action[Client.ChatKeys.Expression] as! String
     message.actionType = ActionType.answer.rawValue
    message.answerData = AnswerAction(action: action)
  } else if type == ActionType.map.rawValue {
    message.actionType = ActionType.map.rawValue
    message.mapData = MapAction(action: action)
  } else if type == ActionType.anchor.rawValue {
    message.actionType = ActionType.anchor.rawValue
    message.anchorData = AnchorAction(action: action)
    message.message = message.anchorData!.text
  }
}

Since, the response from the server for a particular query might contain numerous action types, we create loop inside a method to capture all those action types and save each one of them. Since, there are multiple action types thus we need a list containing all the messages created for the action types. For each action in the loop, corresponding data is saved into their specific objects.

Let’s discuss the individual action objects now.

  • AnswerAction
class AnswerAction: Object {
  dynamic var expression: String = ""
  convenience init(action: [String : AnyObject]) {
    self.init()
    if let expression = action[Client.ChatKeys.Expression] as? String {
      self.expression = expression
    }
  }
}

 This is the simplest action type implementation. It contains a single property `expression` which is a string type. For initializing it, we take the action object and extract the expression key-value and save it.

if type == ActionType.answer.rawValue {
  message.message = action[Client.ChatKeys.Expression] as! String
  message.actionType = ActionType.answer.rawValue
  // pass action object and save data in `answerData`
  message.answerData = AnswerAction(action: action)
}

Above is the way an answer action is checked and data saved inside the `answerData` variable.

2)   MapAction

class MapAction: Object {
  dynamic var latitude: Double = 0.0
  dynamic var longitude: Double = 0.0
  dynamic var zoom: Int = 13

  convenience init(action: [String : AnyObject]) {
    self.init()
    if let latitude = action[Client.ChatKeys.Latitude] as? String,
    let longitude = action[Client.ChatKeys.Longitude] as? String,
    let zoom = action[Client.ChatKeys.Zoom] as? String {
      self.longitude = Double(longitude)!
      self.latitude = Double(latitude)!
      self.zoom = Int(zoom)!
    }
  }
}

This action implementation contains three properties, `latitude` `longitude` `zoom`. Since the server responds the values inside a string, each of them need to be converted to their respective type using force-casting. Default values are provided for each property in case some illegal value comes from the server.

3)   AnchorAction

class AnchorAction: Object {
  dynamic var link: String = ""
  dynamic var text: String = ""

  convenience init(action: [String : AnyObject]) {
    self.init()if let link = action[Client.ChatKeys.Link] as? String,
    let text = action[Client.ChatKeys.Text] as? String {
      self.link = link
      self.text = text
    }
  }
}

Here, the link to the openstreetmap website is saved in order to retrieve the image for displaying.

Finally, we need to call the API and create the message object and use the `write` clock of a realm instance to save it into the DB.

if success {
  self.collectionView?.performBatchUpdates({
    for message in messages! {
    // real write block
      try! self.realm.write {
        self.realm.add(message)
        self.messages.append(message)
        let indexPath = IndexPath(item: self.messages.count - 1, section: 0)
        self.collectionView?.insertItems(at: [indexPath])
      }
   }
}, completion: { (_) in
    self.scrollToLast()
  })
}

list of message items and inserted into the collection view.Below is the output of the Realm Browser which is a UI for viewing the database.

References:

Continue Reading Save Chat Messages using Realm in SUSI iOS