Shifting from Java to Kotlin in SUSI Android

SUSI Android (https://github.com/fossasia/susi_android) is written in Java. After the announcement of Google to officially support Kotlin as a first class language for Android development we decided to shift to Kotlin as it is more robust and code friendly than Java.

Advantages of Kotlin over Java

  1. Kotlin is a null safe language. It changes all the instances used in the code to non nullable type thus it ensures that the developer don’t get any nullPointerException.
  2. Kotlin provides the way to declare Extensive function similar to that of C#. We can use this function in the same way as we use the member functions of our class.
  3. Kotlin also provides support for Lambda function and other high order functions.

For more details refer to this link.

After seeing the above points it is now clear that Kotlin is much more effective than Java and there is harm in switching the code from Java to Kotlin. Lets now see the implementation in Susi Android.

Implementation in Susi Android

In the Susi Android App we are implementing the MVP design with Kotlin. We are converting the code by one activity each time from java to Kotlin. The advantage here with Kotlin is that it is totally compatible with java at any time. Thus allowing the developer to change the code bit by bit instead of all at once.Let’s now look at SignUp Activity implementation in Susi Android.

The SignUpView interface contains all the function related to the view.

interface ISignUpView {


  fun alertSuccess()

  fun alertFailure()

  fun alertError(message: String)

  fun setErrorEmail()

  fun setErrorPass()

  fun setErrorConpass(msg: String)

  fun setErrorUrl()

  fun enableSignUp(bool: Boolean)

  fun clearField()

  fun showProgress()

  fun hideProgress()

  fun passwordInvalid()

  fun emptyEmailError()

  fun emptyPasswordError()

  fun emptyConPassError()


}

The SignUpActivity implements the view interface in the following way. The view is responsible for all the interaction of user with the UI elements of the app. It does not contain any business logic related to the app.

class SignUpActivity : AppCompatActivity(), ISignUpView {


  var signUpPresenter: ISignUpPresenter? = null

  var progressDialog: ProgressDialog? = null


  override fun onCreate(savedInstanceState: Bundle?) {

      super.onCreate(savedInstanceState)

      setContentView(R.layout.activity_sign_up)

      addListeners()

      setupPasswordWatcher()


      progressDialog = ProgressDialog(this@SignUpActivity)

      progressDialog?.setCancelable(false)

      progressDialog?.setMessage(this.getString(R.string.signing_up))


      signUpPresenter = SignUpPresenter()

      signUpPresenter?.onAttach(this)

  }


  fun addListeners() {

      showURL()

      hideURL()

      signUp()

  }


  override fun onOptionsItemSelected(item: MenuItem): Boolean {

      if (item.itemId == android.R.id.home) {

          finish()

          return true

      }

      return super.onOptionsItemSelected(item)

  }

Now we will see the implementation of models in Susi Android in Kotlin and compare it with Java.

Lets First see the implementation in Java

public class WebSearchModel extends RealmObject {

  private String url;

  private String headline;

  private String body;

  private String imageURL;


  public WebSearchModel() {

  }


  public WebSearchModel(String url, String headline, String body, String imageUrl) {

      this.url = url;

      this.headline = headline;

      this.body = body;

      this.imageURL = imageUrl;

  }


  public void setUrl(String url) {

      this.url = url;

  }


  public void setHeadline(String headline) {

      this.headline = headline;

  }


  public void setBody(String body) {

      this.body = body;

  }


  public void setImageURL(String imageURL) {

      this.imageURL = imageURL;

  }


  public String getUrl() {

      return url;

  }


  public String getHeadline() {

      return headline;

  }


  public String getBody() {

      return body;

  }


  public String getImageURL() {

      return imageURL;

  }

}
open class WebSearchModel : RealmObject {


  var url: String? = null


  var headline: String? = null


  var body: String? = null


  var imageURL: String? = null


  constructor() {}


  constructor(url: String, headline: String, body: String, imageUrl: String) {

      this.url = url

      this.headline = headline

      this.body = body

      this.imageURL = imageUrl

  }

}

You can yourself see the difference and how easily with the help of Kotlin we can reduce the code drastically.

For diving more into the code, we can refer to the GitHub repo of Susi Android (https://github.com/fossasia/susi_android).

Resources

Continue ReadingShifting from Java to Kotlin in SUSI Android

Implementing Toolbar(ActionBar) in SUSI Android

SUSI is an artificial intelligence for interactive chat bots. The SUSI Android app (https://github.com/fossasia/susi_android) has a toolbar, that allows different user preferences right up in the menu option. These include functionalities such as search, login, logout and rating the app. The material design toolbar provides a clean and efficient way to do these functions. We will see how we implemented this in the SUSI Android app.

To implement ActionBar in the Android app we will start with adding the dependency below to the gradle file.

compile `com.android.support:appcompat-v7:22.2.0` 

For more information regarding setting up ActionBar in your own project please refer to this link.

Now we will add the Toolbar in our XML file as shown below.

<android.support.v7.widget.Toolbar xmlns:app="http://schemas.android.com/apk/res-auto"

  style="@style/Theme.AppCompat"

  xmlns:android="http://schemas.android.com/apk/res/android"

  android:layout_width="match_parent"

  android:layout_height="wrap_content"

  android:background="?attr/colorPrimary"

  android:minHeight="@dimen/abc_action_bar_default_height_material">


  <ImageView

      android:id="@+id/toolbar_img"

      android:layout_width="@dimen/toolbar_logo_width"

      android:layout_height="@dimen/toolbar_logo_height"

      android:layout_gravity="center"

      app:srcCompat="@drawable/ic_susi_white" />


</android.support.v7.widget.Toolbar>

On Adding the above code we can see the preview of the Toolbar which appears like this.

 

Now we will see how to implement the menu options. In the menu_main.xml we can add options to be shown in the toolbar.

<menu xmlns:android="http://schemas.android.com/apk/res/android"

  xmlns:app="http://schemas.android.com/apk/res-auto"

  xmlns:tools="http://schemas.android.com/tools"

  tools:context=".activities.MainActivity">

  <item

      android:id="@+id/action_search"

      android:icon="@android:drawable/ic_menu_search"

      android:orderInCategory="200"

      android:title="@string/search"

      app:actionViewClass="android.support.v7.widget.SearchView"

      app:showAsAction="always" />

  <item

      android:id="@+id/down_angle"

      android:icon="@drawable/ic_arrow_down_24dp"

      android:title="Search Down"

      android:visible="false"

      app:showAsAction="always"/>

  <item

      android:id="@+id/up_angle"

      android:icon="@drawable/ic_arrow_up_24dp"

      android:title="Search Up"

      android:visible="false"

      app:showAsAction="always"/>

  <item

      android:id="@+id/action_settings"

      android:orderInCategory="100"

      android:title="@string/action_settings"

      app:showAsAction="never" />

  <item

      android:id="@+id/wall_settings"

      android:orderInCategory="100"

      android:title="@string/action_wall_settings"

      app:showAsAction="never" />

  <item

      android:id="@+id/action_share"

      android:orderInCategory="101"

      android:title="@string/action_share"

      app:showAsAction="never" />

We can define different items in the ActionBar as shown above. Also, we have the option by which we can set the visibility of the items. If the visibility of the item is false, then it will be shown in the overflow menu like this.

Now we will see how we can add functionalities to the options present in the menu ActionBar. To add the listeners to the options we have to override the function onCreateOptionsMenu and there we can add the logic for different options as shown below.

@Override

  public boolean onCreateOptionsMenu(final Menu menu) {

      this.menu = menu;

      getMenuInflater().inflate(R.menu.menu_main, menu);

      if(PrefManager.getBoolean(Constant.ANONYMOUS_LOGGED_IN, false)) {

          menu.findItem(R.id.action_logout).setVisible(false);

          menu.findItem(R.id.action_login).setVisible(true);

      } else if(!(PrefManager.getBoolean(Constant.ANONYMOUS_LOGGED_IN, false))) {

          menu.findItem(R.id.action_logout).setVisible(true);

          menu.findItem(R.id.action_login).setVisible(false);

      }


      searchView = (SearchView) MenuItemCompat.getActionView(menu.findItem(R.id.action_search));

      final EditText editText = (EditText)searchView.findViewById(android.support.v7.appcompat.R.id.search_src_text);

      searchView.setOnSearchClickListener(new View.OnClickListener() {

          @Override

          public void onClick(View view) {

              toolbarImg.setVisibility(View.GONE);

              ChatMessage.setVisibility(View.GONE);

              btnSpeak.setVisibility(View.GONE);

              sendMessageLayout.setVisibility(View.GONE);

              isEnabled = false;

          }

      });

}

For diving more into the code, we can refer to the GitHub repo of Susi Android (https://github.com/fossasia/susi_android).

Resources

Continue ReadingImplementing Toolbar(ActionBar) in SUSI Android

Custom UI Implementation for Web Search and RSS actions in SUSI iOS Using Kingfisher for Image Caching

The SUSI Server is an AI powered server which is capable of responding to intelligent answers based on user’s queries. The queries to the susi server are obtained either as a websearch using the application or as an RSS feed. Two of the actions are websearch and RSS. These actions as the name suggests respond to queries based on search results from the web which are rendered in the clients. In order to use use these action types and display them in the SUSI iOS client, we need to first parse the actions looking for these action types and then creating a custom UI for them to display them.

To start with, we need to make send the query to the server to receive an intelligent response from the server. This response is parsed into different action types supported by the server and saved into relevant objects. Here, we check the action types by looping through the answers array containing the actions and based on that, we save the data for that action.

if type == ActionType.rss.rawValue {
   message.actionType = ActionType.rss.rawValue
   message.rssData = RSSAction(data: data, actionObject: action)
} else if type == ActionType.websearch.rawValue {
   message.actionType = ActionType.websearch.rawValue
   message.message = action[Client.ChatKeys.Query] as? String ?? ""
}

Here, we parsed the data response from the server and looked for the rss and websearch action type followed by which we saved the data we received from the server for each of the action types in their own objects.

Next, when a message object is created, we insert it into the dataSource item by appending it and use the `insertItems(at: [IndexPath])` method of collection view to insert them into the views at a particular index. Before adding them, we need to create a Custom UI for them. This UI will consist of a Collection View which is scrollable in the horizontal direction inside a CollectionView Cell. To start with this, we create a new class called `WebsearchCollectionView` which will be a `UIView` consisting of a `UICollectionView`.

We start by adding a collection view into the UIView inside the `init` method by overriding it. Declare a collection view using flow layout and scroll direction set to `horizontal`. Also, hide the scroll indicators and assign the delegate and datasource to `self`.

Now to populate this collection view, we need to specify the number of items that will show up. For this, we make use of the `message` variable declared. We use the `websearchData` in case of websearch action and `rssData` otherwise.

Now to specify the number of cells, we use the below method which returns the number of rss or websearch action objects and defaults to 0 such cells.

func collectionView(_ collectionView: UICollectionView, numberOfItemsInSection section: Int) -> Int {
    if let rssData = message?.rssData {
        return rssData.count
    } else if let webData = message?.websearchData {
        return webData.count
    }
    return 0
}

We display the title, description and image for each object for which we need to create a UI for the cells. Let’s start by creating a Custom Collection View cell with the imageview and 2 labels for title and description.

The imageview is given a contentMode of `aspectFit` and assigned a placeholder image in case the image doesn’t exist. The title and description labels are assigned the same font size, the title being bolder and both are center aligned.

class WebsearchCell: BaseCell {

   var imageView: UIImageView = {
       let iv = UIImageView()
       iv.contentMode = .scaleAspectFit
       iv.image = UIImage(named: "placeholder")
       return iv
   }()

   let titleLabel: UILabel = {
       let label = UILabel()
       label.textColor = .black
       label.font = UIFont.boldSystemFont(ofSize: 14)
       label.textAlignment = .center
       label.numberOfLines = 2
       label.backgroundColor = Color.grey.lighten3
       return label
   }()

   let descriptionLabel: UILabel = {
       let label = UILabel()
       label.font = UIFont.systemFont(ofSize: 14)
       label.textAlignment = .center
       return label
   }()

}

Next, we add constraints for each such view adding a title and description label adding a small margin on both sides of the cell for a cleaner UI.

addSubview(imageView)
addSubview(titleLabel)
addSubview(descriptionLabel)
descriptionLabel.numberOfLines = 5
addConstraintsWithFormat(format: "H:|-4-[v0(\(frame.width * 0.4))]-4-[v1]-4-|", views: imageView, titleLabel)
addConstraintsWithFormat(format: "|-\(frame.width * 0.4 + 8)-[v0]-4-|", views: descriptionLabel)
addConstraintsWithFormat(format: "V:|-4-[v0]-4-|", views: imageView)
addConstraintsWithFormat(format: "V:|-4-[v0(44)]-4-[v1]-4-|", views: titleLabel, descriptionLabel)

Now to use this custom cell, we first need to register it with the collection view and then we can use it easily in the `cellForItemAt` method. Since we are using Kingfisher for image caching, we use the `.kf.setImage` method to download the image from a URL and cache it as soon as it is downloaded.

collectionView.register(WebsearchCell.self, forCellWithReuseIdentifier: cellId)
 func collectionView(_ collectionView: UICollectionView, cellForItemAt indexPath: IndexPath) -> UICollectionViewCell {
       if let cell = collectionView.dequeueReusableCell(withReuseIdentifier: cellId, for: indexPath) as? WebsearchCell {
           cell.backgroundColor = .white

           if message?.actionType == ActionType.rss.rawValue {
               let feed = message?.rssData?.rssFeed[indexPath.item]
               cell.titleLabel.text = feed?.title
               cell.descriptionLabel.text = feed?.desc                cell.imageView.kf.setImage(with: URL(string: feed?.rssData?.image))
           } else if message?.actionType == ActionType.websearch.rawValue {
               let webData = message?.websearchData[indexPath.item]
               cell.titleLabel.text = webData?.title
               cell.descriptionLabel.text = webData?.desc.html2String                cell.imageView.kf.setImage(with: URL(string: feed?.webData?.image))
           }
           return cell
       } else {
           return UICollectionViewCell()
       }
   }

We check the action type and assign data based on that. Also, for the images, if they don’t exist a placeholder is added.

Since we are done with the Custom UI of the cell, we need to add it to the chat cell. For that, we add this `UIView` as a subview. For reasons of reusability, the cell was extracted into a separate one and call the `prepareForReuse()` method for reusability. This is followed by adding a subview and setting constraints for each cell and assigning the message object.

class RSSCell: ChatMessageCell {

   var message: Message? {
       didSet {
           self.addWebsearchView()
       }
   }

   lazy var websearchView: WebsearchCollectionView = {
       let view = WebsearchCollectionView()
       return view
   }()

   override func setupViews() {
       super.setupViews()
       prepareForReuse()
   }

   func addWebsearchView() {
       self.addSubview(websearchView)
       self.addConstraintsWithFormat(format: "H:|[v0]|", views: websearchView)
       self.addConstraintsWithFormat(format: "V:[v0(135)]", views: websearchView)
       websearchView.message = message
   }

}
if message.actionType == ActionType.rss.rawValue || message.actionType == ActionType.websearch.rawValue {
   if let cell = collectionView.dequeueReusableCell(withReuseIdentifier: ControllerConstants.rssCell, for: indexPath) as? RSSCell {
       cell.message = message
       return cell
   } else {
       return UICollectionViewCell()
   }
}

This is all we need to add a custom UI to a chat cell in SUSI iOS, very simple and clean.

Check the screenshot below, the app in action.

Resources:

Continue ReadingCustom UI Implementation for Web Search and RSS actions in SUSI iOS Using Kingfisher for Image Caching

Hotword Recognition in SUSI iOS

Hot word recognition is a feature by which a specific action can be performed each time a specific word is spoken. There is a service called Snowboy which helps us achieve this for various clients (for ex: iOS, Android, Raspberry pi, etc.). It is basically a DNN based hotword recognition toolkit.

In this blog, we will learn how to integrate the snowboy hotword detection wrapper in the SUSI iOS client. This service can be used in any open source project but for using it commercially, a commercial license needs to be obtained.

Following are the files that need to be added to the project which are provided by the service itself: snowboy-detect.h libsnowboy-detect.a and a trained model file which can be created using their online service: snowboy.kitt.ai. For the sake of this blog, we will be using the hotword “Susi”, the model file can be found here.

The way how snowboy works is that speech is recorded for a few seconds and this data is detected with an already trained model by a specific hotword, now if snowboy returns a 1 means word has been successfully detected else wasn’t.

We start with creation of a wrapper class in Objective-C which can be found wrapper and the bridging header in case this needs to be added to a Swift project. The wrapper contains methods for setting sensitivity, audio gain and running the detection using the buffer. It is a wrapper class built on top of the snowboy-detect.h header file.

Let’s initialize the service and run it. Below are the steps followed to enable hotword recognition and print out whether it successfully detected the hotword or not:

  • Create a ViewController class with extensions
    • AVAudioRecorderDelegate
    • AVAudioPlayerDelegate

since we will be recording speech.

  • Import AVFoundation
  • Create a basic layout containing a label which detects whether hotword detected or not and create corresponding `IBOutlet` in the ViewController and a button to trigger the start and stop of recognition.
  • Create the following variables:
    • let WAKE_WORD = “Susi” // hotword used
    • let RESOURCE = Bundle.main.path(forResource: “common”, ofType: “res”)
    • let MODEL = Bundle.main.path(forResource: “susi”, ofType: “umdl”) //path where the model file is stored
    • var wrapper: SnowboyWrapper! = nil // wrapper instance for running detection
    • var audioRecorder: AVAudioRecorder! // audio recorder instance
    • var audioPlayer: AVAudioPlayer!
    • var soundFileURL: URL! //stores the URL of the temp reording file
    • var timer: Timer! //timer to fire a function after an interval
    • var isStarted = false // variable to check if audio recorder already started
  • In `viewDidLoad` initialize the wrapper and set sensitivity and audio gain. Recognition best happens when sensitivity is set to `0.5` and audio gain is set to `1.0` according to the docs.
override func viewDidLoad() {
    super.viewDidLoad()
    wrapper = SnowboyWrapper(resources: RESOURCE, modelStr: MODEL)
    wrapper.setSensitivity("0.5")
    wrapper.setAudioGain(1.0)
}
  • Create an `IBAction` for the button to start recognition. This action will be used to start or stop the recording in which the action toggles based on the `isStarted` variable. When true, recording is stopped and the timer invalidated else a timer is started which calls the `startRecording` method with an interval of 4 seconds.
@IBAction func onClickBtn(_ sender: Any) {
  if (isStarted) {
    stopRecording()
    timer.invalidate()
    btn.setTitle("Start", for: .normal)
    isStarted = false
  } else {
    timer = Timer.scheduledTimer(timeInterval: 4, target: self, 
    selector: #selector(startRecording), userInfo: nil, repeats: true)
    timer.fire()
    btn.setTitle("Stop", for: .normal)
    isStarted = true
  }
}
  • Next, we add the start and stop recording methods.
    • First, a temp file is created which stores the recorded audio output
    • After which, necessary record configurations are made such as setting the sampling rate.
    • The recording is then started and the output stored in the temp file.
func startRecording() {
  do {
    let fileMgr = FileManager.default
    let dirPaths = fileMgr.urls(for: .documentDirectory, in: .userDomainMask)
    soundFileURL = dirPaths[0].appendingPathComponent("temp.wav")
    let recordSettings = [AVEncoderAudioQualityKey: 
    AVAudioQuality.high.rawValue,
    AVEncoderBitRateKey: 128000,
    AVNumberOfChannelsKey: 1,
    AVSampleRateKey: 16000.0] as [String : Any]
    let audioSession = AVAudioSession.sharedInstance()
    try audioSession.setCategory(AVAudioSessionCategoryRecord)
    try audioRecorder = AVAudioRecorder(url: soundFileURL,
settings: recordSettings as [String : AnyObject])
    audioRecorder.delegate = self
    audioRecorder.prepareToRecord()
    audioRecorder.record(forDuration: 2.0)
    instructionLabel.text = "Speak wake word: \(WAKE_WORD)"print("Started recording...")
  } catch let error {
    print("Audio session error: \(error.localizedDescription)")
  }
}
  • The stop recording method, stops the audioRecorder instance and updates the instruction label to show the same.
func stopRecording() {
  if (audioRecorder != nil && audioRecorder.isRecording) {
    audioRecorder.stop()
  }
  instructionLabel.text = "Stop"
  print("Stopped recording...")
}

The final recognition is done in the `audioRecorderDidFinishRecording` delegate method which runs the snowboy detection function which processes the audio recording in the temp file by creating a buffer and storing the audio in it and giving the wrapper this buffer as input which processes the buffer and returning a `1` is hotword was successfully detected.

func runSnowboy() {
  let file = try! AVAudioFile(forReading: soundFileURL)
  let format = AVAudioFormat(commonFormat: .pcmFormatFloat32, 
  sampleRate: 16000.0, channels: 1, interleaved: false)
  let buffer = AVAudioPCMBuffer(pcmFormat: format, frameCapacity: AVAudioFrameCount(file.length))
  try! file.read(into: buffer)
  let array = Array(UnsafeBufferPointer(start: 
  buffer.floatChannelData![0], count:Int(buffer.frameLength)))
  // print output
  let result = wrapper.runDetection(array, length: Int32(buffer.frameLength))
  print("Result: \(result)")
}

To test this out, click the start button and speak different words and you will notice that once the Hot Word is spoken, log with `result: 1` is printed out.

The snowboy hotword recognition also offers to train the personalized model with the help of Rest Apis for which the docs can be found here. The complete project implementation can be found here.

Sources:

Continue ReadingHotword Recognition in SUSI iOS

Showing RSS and Table Type Replies in SUSI Messenger Bots

All the messengers have a “plain text” reply support. To show web search (RSS) or table type replies, either:

  1. We need a “list type” (as in Facebook messenger) or “table type” reply support built in the messenger itself.
                                                                  or
  2. We need to convert the RSS or table type reply by SUSI API to plain text, so that we can send it, due to the “plain text” reply support available in almost every messenger.

The second point is the most favorable approach, as that way, replying with RSS or table type results is dependent only on the text support feature in the messenger. This way RSS or table type replies can be shown in messengers like REST API Gitter (which do not provide any other reply type support except text).

In SUSI web app, the UI of the web search and table type results:

As the SUSI web app is made in React js, it provided the app with necessary features to show the results this way. The messengers may not be having these required features.

So the task is we need to convert the RSS or table type replies by SUSI API to plain text to send it to the messenger.

Let’s work it out.

Converting RSS results to text:

First get familiar with the SUSI API reply to query “why” by visiting this url – http://api.susi.ai/susi/chat.json?q=why.

The JSON object returned will be constituting an array of objects as the value of the data key like:

"data": [
        {
        "title": "Why is Oracle male?",
        "description": "Why is Oracle male?. http dba oracle com why is male htm. Oracle Why is masculine?. ",
        "link": "http://dba-oracle.com/t_why_is_oracle_male.htm"
      }
]

If we notice carefully each object has 3 main keys namely “title”, “description” and “link”. So extracting these 3 properties from each object and binding them together into 1 string is the task we need to do.

So we traverse each object (i.e. rss result) in the data array and we keep on appending the values of “title”, “description” and “link” key values to the ans variable. At the end we send this resultant string to the messenger bot as a reply.  

Suppose we have the returned JSON object, in the data variable.

// storing the number of rss results
var metaCnt = data.answers[0].metadata.count;
    for(var i=0;i<metaCnt;i++){
        ans += ('Title : ');
ans += data.answers[0].data[i].title+', ';
        ans += ('Link : ');
        ans += data.answers[0].data[i].link+', ';
        ans += '\n\n';
    }

// send the message in ans variable to the messenger

Converting table type replies to text:

First get familiar with the SUSI API reply to query “why” by visiting this url – http://api.susi.ai/susi/chat.json?q=universities+in+australia.

The JSON object returned will be constituting an array of objects representing universities as the value of the data key in this form:

{
    "alpha_two_code": "AU",
    "name": "Australian Correspondence Schools",
    "country": "Australia",
    "web_page": "http://www.acs.edu.au/"
}

Here too, each object has 3 main keys namely “name”, “country” and “web_page”. So extracting these 3 properties from each object and binding them together into 1 string can make the things work.

Again traverse each object (i.e. university object) in the data array and we keep on appending the values of “name”, “country” and “web_page” key values to the ans variable. At the end we send this resultant string to the messenger bot as a reply.

Suppose we have the returned JSON object in the data variable.

    // the 3 main columns which we need are stored in colNames variable
    var colNames = data.answers[0].actions[0].columns;
    
    // storing the number of table entries
    var metaCnt = data.answers[0].metadata.count;
    for(var i=0;i<metaCnt;i++){
        for(var cN in colNames){
            // The column name
            ans += (colNames[cN]+' : ');
            // value for that column name
            ans += data.answers[0].data[i][cN]+', ';
        }
        ans += '\n\n';
    }

    // send the message in ans variable to the messenger

Resources

  1. By Slobodan Stojanović from smashing magazineDevelop a chat bot with node js.
  2. By Mikhail Larionov from Facebook blogsList templates and check box plugin
Continue ReadingShowing RSS and Table Type Replies in SUSI Messenger Bots

Using Templates in SUSI FBbot

The SUSI AI Fbbot previously showed rss and table type replies as plain text to the user. To enhance the user experience, Facebook provides with templates which can be used in it’s messenger. Using those, we show rss and table type replies wrapped up in a better U.I. creating a better user experience.

The 4 basic template structures that can be used for this task are:

  1. Button template
  2. Generic template
  3. List template
  4. Receipt template

List template is used in SUSI A.I. Fbbot because rss reply and table type reply both are lists of data.
The basic syntax for list template with reference to Fb official docs is:

"message": {
    "attachment": {
        "type": "template",
        "payload": {
            "template_type": "list",
            "top_element_style": "compact",
            "elements": [
                {
                    "title": "Classic White T-Shirt",
                    "subtitle": "100% Cotton, 200% Comfortable",
                    "default_action": {
                        "type": "web_url",
                        "url": "https://peterssendreceiveapp.ngrok.io/view?item=100"
                    },
                    "buttons": [
                        {
                            "title": "Buy",
                            "type": "web_url",
                            "url": "https://peterssendreceiveapp.ngrok.io/shop?item=100"                     
                        }
                    ]                
                }
            ]
        }
    }
}

This code shows a reply to the user like this:

If we want to show a “View more” button at the end of the list, we can add a “buttons” key and an array as its value which will have information regarding all the buttons we want to show.

The code below will show a “View more” button at the end of the list:

"elements": [
            {
                // all the elements, like shirt in the above case               
            }
       ],
       "buttons": [
            {
                "title": "View More",
                "type": "postback",
                "payload": "payload"                        
               }
       ]

Let’s learn how to incorporate these features to rss results in SUSI Fbbot:

When sending a reply using list template, “elements” key must have an array type value which will be constituted of list items. Therefore, we need to push all the rss results into that array and set it as the value of the “elements” key.

Let’s develop the code part:

Fetch the JSON object from SUSI API url with query as “why” i.e. http://api.susi.ai/susi/chat.json?q=why. Let body be a variable which stores the returned JSON object.

  • The below code fills up an array (namely elementsVal) with all the rss results:
var elementsVal = [];
var metaCnt = body.answers[0].metadata.count;
for(var i=0;i<((metaCnt>4)?4:metaCnt);i++){
    elementsVal.push(
        {
            "title": body.answers[0].data[i].title,
            "subtitle": body.answers[0].data[i].link
        }
    );
}
  • Setting the elementsVal array as the value of the “elements” key:
var message = {
    "type": "template",
    "payload": 
    {
        "template_type": "list",
        "top_element_style": "compact",
        "elements": elementsVal
    }
};
  • Sending this message as a reply to the user:
sendTextMessage(sender, message);

Same procedure can be used to render table type replies in the bot using the list template.

Generic template provided by Facebook can also used to render the web and table type replies. This template helps in showing the results in square boxes, which can be changed using left or right arrows.

Resources:

  1. By Mikhail Larionov from Facebook blogsList templates and check box plugin
  2. By Slobodan Stojanović from smashing magazineDevelop a chat bot with node js.
Continue ReadingUsing Templates in SUSI FBbot

Integration of SUSI AI to Facebook

It’s easy to create your own SUSI AI Facebook messenger bot. You can read the official documentation by Facebook to get started.

Messenger bots use a web server to send and receive messages. You also need to have the bot be authenticated to speak with the web server and be approved by Facebook before getting public.

If any problems faced, visit the susi_fbbot repository which hosts the code for SUSI Facebook Messenger bot.

We will be using Node js technology to develop the FB bot. First, let’s see on how to request an answer from the SUSI API.

To call Susi API and fetch an answer from it for a query (‘hi’ in this case). Let’s first visit http://api.asksusi.com/susi/chat.json?q=hi from the browser. We will get a JSON object as follows:

The answer can be found as the value of the key named expression. In this case, it is “Hallo!”.

To fetch the answer through coding, we can use this code snippet in Node js:

// including request module
var request = require(‘request’);

// setting options to make a successful call to Susi API.
var options = {
            method: 'GET',
            url: 'http://api.asksusi.com/susi/chat.json',
            qs: 
            {
                timezoneOffset: '-330',
                q: 'hi'
            }
        };

// A request to the Susi bot
request(options, function (error, response, body) {
    if (error)
        throw new Error(error);
    // answer fetched from susi
    ans = (JSON.parse(body)).answers[0].actions[0].expression;
}

The properties required for the call are set up through a JSON object (i.e. options). Pass the options object to our request function as its 1st parameter. The response from the API will be stored in ‘body’ variable. We need to parse this body, to be able to access the properties of that body object. Hence, fetching the answer from Susi API.

Now let’s dive into the code of receiving and messaging back to the user on Facebook:

  • A generic function to send a message to the user.
// the first argument is the sender id and the second is the text to send.
function sendTextMessage(sender, text) {
    var messageData = { text:text };
    
    // making a post request to facebook graph API to send message.
    request({
        url: 'https://graph.facebook.com/v2.6/me/messages',
        qs: {access_token:token},
        method: 'POST',
        json: {
            recipient: {id:sender},
            message: messageData,
        }
    }, function(error, response, body) {
        if (error) {
            console.log('Error sending messages: ', error);
        } else if (response.body.error) {
            console.log('Error: ', response.body.error);
        }
    });
}
  • According to step 9, in the below instructions we need to include this code snippet too:
// for facebook verification
app.get('/webhook/', function (req, res) {
    if (req.query['hub.verify_token'] === 'my_voice_is_my_password_verify_me') {
        res.send(req.query['hub.challenge']);
    }
    res.send('Error, wrong token');
});
  • When user messages to our bot, we need to extract the text of the message from the request body. Then we need to extract the reply from the SUSI API and send it back to the user.
// when a message from a user is received.
app.post('/webhook/', function (req, res) {
    var messaging_events = req.body.entry[0].messaging
    for (var i = 0; i < messaging_events.length; i++) {
        // fetching the current event
        var event = req.body.entry[0].messaging[i];

        // fetching the sender id
        var sender = event.sender.id;
        
        // if the event is a message event
        if (event.message && event.message.text) {
            var text = event.message.text;

            // Construct the query for susi
            var queryUrl = 'http://api.susi.ai/susi/chat.json?q='+encodeURI(text);
            var message = '';

            // Wait until done then reply
            request({
                url: queryUrl,
                json: true
            }, function (error, response, body) {
                if (!error && response.statusCode === 200) {
              // fetch the answer from the response body and save it in message variable.
                    // send the reply
                    sendTextMessage(sender, message);
                } 
                
            // if, due some reasons the answer couldn’t be fetched
            else {
                    message = 'Oops, Looks like Susi is taking a break, She will be back soon';
                    sendTextMessage(sender, message);
                }
            });
        }
        if (event.postback) {
            var text = JSON.stringify(event.postback);
            sendTextMessage(sender, "Postback received: "+text.substring(0, 200), token);
            continue;
        }
    }
    res.sendStatus(200)
})

Upload the code developed to a repository.

Let’s follow the below steps, to achieve a working fb messenger bot:

  1. Create a Facebook page here.

    Creating a FB Page
    New FB Page

    1. Create a new Heroku app here.

    New Heroku App

    1. Connect the Heroku app to the repository hosting our code.

    Connect to Github

    4. Deploy on the development branch. If you intend to contribute, it is recommended to Enable Automatic Deploys.

    Branch Deployment

    Successful Deployment

    5. Create or configure a Facebook App or Page here.

    New FB App

    6. Get started with Messenger tab in the created app.


    7.
    In the Page Access Token select the FB page that you created and generate the token and save it somewhere for future use.

    Token Generation

    8. Now, go to the Heroku app, select the settings tab and add the environment variable as shown, where the key is FB_PAGE_ACCESS_TOKEN and value is the token generated in the previous step.

    9. Create a web hook on the facebook app dashboard. The Callback URL should be https://<your_app_name>.herokuapp.com/webhook/ and rest should be as shown in the image below.

    App Complete

    10. Go to Terminal and type in this command to trigger the Facebook app to send messages. Remember to use the token you requested earlier.
    “`
    curl -X POST “https://graph.facebook.com/v2.6/me/subscribed_apps?access_token=<PAGE_ACCESS_TOKEN>”
    “`

    11. Go to the Facebook page created and locate ‘Message Now’ or go to https://m.me/PAGE_USERNAME

    12. Enjoy chatting with Susi.

    Resources

Continue ReadingIntegration of SUSI AI to Facebook

Integration of SUSI AI in Twitter

We will be making a Susi messenger bot on Twitter. The messenger bot will tweet back to your tweets and reply instantly when you chat with it. Feel free to tweet to the already made SUSI AI account (mentioning @SusiAI1 in it). Follow it, to have a personal chat.

Make a new account, which you want to use as the bot account. You can make one from sign up option from https://www.twitter.com.

Prerequisites

To create your account on -:
1. Twitter
2. Github
3. Heroku
4. Node js

Setup your own Messenger Bot

1. Make a new app here, to know the access token and other properties for our application. These properties will help us communicate with Twitter.

Click “modify the app permissions” link, as shown here:

Select the Read, Write and Access direct messages option:

Don’t forget to click the update settings button at the bottom.

Click the Generate My Access Token and Token Secret button.

3. Create a new heroku app here.

This app will accept the requests from Twitter and Susi api.

4. Create a config variable by switching to settings page of your app.
  
  The name of your first config variable should be HEROKU_URL and its value is the url address of the heroku app created by you.
 

The other config variables that need to be created will be these:
 

The corresponding names of these variables in the same order are:
  i) Access token
  ii) Access token secret
  iii) Consumer key
  iv) Consumer secret
  
We need to visit our app from here, the keys and access tokens tab will help us with the values of these variables.

  1. Let’s start with the code part of the integration of SUSI AI to Twitter. We will be using Node js to achieve this integration.

First we need to require some packages:

Now using the Twit module, we need to authenticate our requests, by using our environment variables as set up in step 4:

Now let’s make a user stream:

var stream = T.stream('user');

We will be using the capabilities of this stream, to catch events of getting tweeted or receiving a direct message by using:

stream.on('tweet', functionToBeCalledWhenTweeted);
stream.on('follow', functionToBeCalledWhenFollowed);
stream.on('direct_message', functionToBeCalledWhenDirectMessaged);

So, when a person tweets to our account like this:

We can catch it with ‘tweet’ event and execute a set of instructions:

stream.on('tweet', tweetEvent);

    function tweetEvent(eventMsg) {
        var replyto = eventMsg.in_reply_to_screen_name;     

       // to store the message tweeted excluding '@SusiAI1' substring
        var text = eventMsg.text.substring(9);

        // to store the name of the tweeter
        var from = eventMsg.user.screen_name;
        
        if (replyto === 'SusiAI1') {
            var queryUrl = 'http://api.asksusi.com/susi/chat.json?q=' + encodeURI(text);
            var message = '';
            request({
                url: queryUrl,
                json: true
            }, function (err, response, data) {
                if (!err && response.statusCode === 200) {
                    // fetching the answer from the data object returned
                                        message = data.answers[0].actions[0].expression + data;


                }
                else {
                    message = 'Oops, Looks like Susi is taking a break';    
                    console.log(err);
                }
                console.log(message);
                // If the message length is more than tweet limit
                if(message.length > 140){
                    tweetIt('@' + from + ' Sorry due to tweet word limit, I have sent you a personal message. Check inbox'+date);
                    sendMessage(from, message);
                }
                else{
                    tweetIt('@' + from + ' ' + message + date);
                }
            });
        }
    }
  • When we a person follows this SUSI AI account, we can thank him/her by making use of the “follow” event. Also, we need to follow him/her back, to enable personal chat between Susi and that person (according to the rules of twitter):
stream.on('follow',followed);

function followed(eventMsg) {
        console.log('Follow event !');
        var name = eventMsg.source.name;
        var screenName = eventMsg.source.screen_name;
        var user_id1 = eventMsg.source.id_str;

        // To follow back the person.
        T.post('friendships/create', {user_id : user_id1},  function(err, tweets, response){
            if (err) {
                console.log("Couldn't follow back!");
            } 
            else {    
tweetIt('@' + screenName + ' Thank you for following me! I followed you back, you can also direct message me now! ');
                console.log("Followed back!");
            } 
        }); 
    }

When we personally message this SUSI AI account

This can be handled through the ‘direct_message’ event:

stream.on('direct_message', reply);
    function reply(directMsg) {
        console.log('You receive a message!');
        // If its our own bot messaging, ignore it, as this can lead to an infinite loop when we answer a user.
        if (directMsg.direct_message.sender_screen_name === 'SusiAI1') {
            return;
        }

        // code to fetch the reply from SUSI API
        
        // reply the user with the SUSI API's message
        sendMessage(directMsg.direct_message.sender_screen_name, message);
        });
    }
  • The tweetIt and sendMessage function code can be seen from the repo code.

6. Connect the heroku app to the forked repository.

Connect the app to Github by selecting the name of this forked repository.

7. Deploy on development branch. If you intend to contribute, it is recommended to Enable Automatic Deploys.

Branch Deployment.

Successful Deployment.

  1. Visit your own personal account and tweet to this new bot account with your query and enjoy a tweet back from the bot account! Also, you can enjoy personal chatting with Susi.

    Feel free to play around with the already made SUSI AI account on twitter here. Follow it, to have a personal chat with it.

    Resources
Continue ReadingIntegration of SUSI AI in Twitter

Integration of Susi AI to Gitter

This blog post discusses the development of Susi Messenger bot on Gitter. It replies instantly to the messages sent to it, using the Susi API. The Streaming API notifies us when a user messages to the SUSI chat room. The REST API helps to message back with a reply from SUSI API, to the SUSI chat room.

Feel free to message to the already made SUSI AI account on Gitter and have a chat with it.

Prerequisites

  1. Basic knowledge about calling API’s and fetching data or posting data to the API.
  2. Node.js language.
  3. Github
  4. Heroku

Figure – Architecture for running SUSI AI on different messaging services.

This blog post will walk you through each of the steps required to integrate SUSI AI to Gitter:

Setup SUSI AI Bot on Gitter

  1. Create a Github or twitter account with a username having ‘Susi’ as its substring because this is the name that will be shown with the reply string we will get from Susi AI.
  2. Now you need to sign in to Gitter with a twitter or Github account from here.
  3. Create your community by visiting this page. After writing your community name press next, invite the people you want to be in this room and press next. You will be redirected to your communities lobby. This lobby is the chat room to which we will deploy our SUSI AI.
  4. Now visit the Gitter developer page, press sign in on the top right. You will be redirected to your apps page. Copy the personal access token written there as shown in this image: (The area colored black will have your access token).

5. On a new tab, in your browser visit   https://api.gitter.im/v1/rooms?access_token=YOUR_ACCESS_TOKEN, with YOUR_ACCESS_TOKEN replaced by the token we just copied.

A JSON object will be shown on our browser screen. You will see the value of ‘name’ key as YOUR_COMMUNITY_NAME/Lobby. Copy the id of this chat room, as we will need it later. You can refer to the image below, you will have your chat room id in the area colored black.

  1. Create a new heroku app here.

This app will accept the requests from Gitter and Susi api.

  1. Set the config variables for this heroku app in the setting tab of your account. Set ROOM_ID to the id of the chat room and TOKEN to the personal access token, we copied in steps 4 and 5.

These were the formalities to be done to have our chat bot account on Gitter.

  1. Let’s jump to the code part of how this integration will be done:

To use the two config variables set in Heroku, we need these two lines in our Node js code:

var roomId = process.env.ROOM_ID;
var token = process.env.TOKEN;

We need to set up an options variable with our access token and room id in it:

// Setting the options variable to use it in the https request block
var options = {
    hostname: 'stream.gitter.im',
    port:     443,
    path:     '/v1/rooms/' + roomId + '/chatMessages',
    method:   'GET',
    headers:  {'Authorization': 'Bearer ' + token}
};

We will send this options variable when making a request so that Gitter lets our request through and notifies us when a client messages to our SUSI chat room.

The res.on(‘data’) accepts a function which is called when a client messages to our SUSI chat room:

// making a request to gitter stream API
var req = https.request(options, function(res) {
 res.on('data', function(chunk) {
    // do stuff
 }
}

req.on('error', function(e) {
 console.log('Hey something went wrong: ' + e.message);
});

req.end();

According to the docs of REST API in Gitter, the JSON data that we receive when a client sends a message to a chat room is like this:

To get this response set in a variable, we can use this code snippet:

res.on('data', function(chunk) {
   var msg = chunk.toString();
   if(msg != " \n"){              // If message is not an empty message
     var jsonMsg = JSON.parse(msg);

Now we have this json response as shown in the above figure in the jsonMsg variable. To extract the client’s message from this json object:

var clientMsg = jsonMsg.text;

As we now have the user query in clientMsg variable. Let’s call Susi API and fetch an answer from it for a query.

As an example, let’s first take the query as ‘hi’ and visit http://api.asksusi.com/susi/chat.json?q=hi from the browser. We will get a JSON object as follows:

{
        "query": "hi",
        "count": 1,
        "client_id": "aG9zdF8xMDMuMjkuMjIyLjE4MA==",
        "query_date": "2017-07-17T02:29:44.171Z",
        "answers": [{
            "data": [{
                "0": "hi",
                "token_original": "hi",
                "token_canonical": "hi",
                "token_categorized": "hi",
                "timezoneOffset": "-330",
                "language": "en"
            }],
            "metadata": {"count": 1},
            "actions": [{
                    "type": "answer",
                    "language": "de",
                    "expression": "Hallo!"
            }],
   "skills": ["/susi_skill_data/models/general/smalltalk/de/German-Standalone-aiml2susi.txt"]
        }],
        "answer_date": "2017-07-17T02:29:44.179Z",
        "answer_time": 8,
        "language": "en",
        "session": {"identity": {
            "type": "host",
            "name": "103.29.222.180",
            "anonymous": true
        }}
    }

The answer can be found as the value of the key named expression. In this case, it is “Hallo!”.

To fetch the answer through coding for our client message, we can use this code snippet in Node js:

// including request module
var request = require(‘request’);

// setting options to make a successful call to Susi API.
var susiOptions = {
            method: 'GET',
            url: 'http://api.asksusi.com/susi/chat.json',
            qs:
            {
                timezoneOffset: '-330',
                q: clientMsg  //the client message sent to SUSI chat room.
            }
        };

// A request to the Susi bot
request(susiOptions, function (error, response, body) {
    if (error)
        throw new Error(error);
    // answer fetched from susi
    ans = (JSON.parse(body)).answers[0].actions[0].expression;
}

The properties required for the call are set up through a JSON object (i.e. susiOptions). Pass the susiOptions object to our request function as its 1st parameter. The response from the API will be stored in ‘body’ variable. We need to parse this body, to be able to access the properties of that body object. Hence, fetching the answer from Susi API.

As we now have the answer, let’s call the API of Gitter to show our answer back to the user. Let’s code the request for that:

// To send a reply by Susi AI to client's message back to Gitter
         var gitterOptions = {
                               method: 'POST',
                               url: 'https://api.gitter.im/v1/rooms/'+roomId+'/chatMessages',
                               headers:
                               {
                                 'authorization': 'Bearer '+ token ,
                                 'content-type': 'application/json',
                                 'accept': 'application/json'
                               },
                               body:
                               {
                                 text: ans
                               },
                               json: true
                             };

         // making the request to Gitter API
         request(gitterOptions, function (error, response, body) {
           if(error)
             throw new Error(error);
           console.log(body);
         });

Hence, we have made the basic chat work!

The streaming API of Gitter notifies us for every message sent to our chat room. We will also be notified about the reply message sent by ourselves. To not fall into an infinite loop of answers and questions by SUSI itself, we must include this line in our code:

res.on('data', function(chunk) {
   var msg = chunk.toString();
   if(msg != ” \n”){              // If message is not an empty message
     var jsonMsg = JSON.parse(msg);
     if(jsonMsg.fromUser.displayName != 'SusiAI'){ // If it’s not our own answer
        // do stuff 
     }
   }
}

req.on('error', function(e) {
 console.log('Hey something went wrong: ' + e.message);
});

req.end();

The display name in my case is ‘SusiAI’, but it may be different in your case according to the Github or Twitter id made by you.

  1. Upload this code to Github.
  2. Connect the Heroku app to the Github repository, which has your code.

  1. Deploy on the development branch. If you intend to contribute, it is recommended to Enable Automatic Deploys.

Branch Deployment.

Successful Deployment.

  1. Go to your Gitter room created and enjoy chatting with Susi.Resources
Continue ReadingIntegration of Susi AI to Gitter