“STOP” action in SUSI Android App

Generally whenever there was a long query asked from SUSI through speech, it would respond to the user with a speech output, similarly the output is given through speech whenever the user clicks on the “Try it” button on the skill details activity.

The long answers for e.g. asking SUSI “How to cook biryani ?” gives a very long response which when conveyed through the speech output takes a long amount of time. Also most of the time users don’t want to listen to such long answers, but at the same time there is no option or feature to stop SUSI. The user either needs to switch over to another activity or has to close the app.

So, to solve this problem such that neither the user has to shut down the app nor the user has to switch over to another activity, the “STOP” action was added in SUSI.

The “STOP” action was integrated in the server and is of following type :

 

“actions”: [{“type”: “stop”}],

How to define STOP response ?

To integrate this action type in the app, a separate response type was added and checked for. In the file ParseSusiResponseHandler.kt the action type stop was added as :

Constant.STOP -> try {
  stop = susiResponse.answers[0].actions[1].type
} catch (e: Exception) {

}

So, now whenever the query of type “stop” was entered by the user it would be caught by this block and further processing could be done. The stop variable takes the value from the JSON response fetched and the value extracted is the one stored against the type key in the second field of the actions JSON in the first field of the answers JSON.

What to be done when executing STOP?

After the STOP action was caught the task to do was to define the behaviour of the app when this response was caught. So, first thing that should happen when STOP is received is that the TextToSpeech Engine should close so that SUSI no longer can speak the sentence but defining top is more than just closing the TTS engine. STOP signifies that any activity that is being continued right now should be stopped.

So, Using the android lifecycle methods I added a function in the IChatView interface to define what actions should take place in the stopping process.

override fun stopMic() {
  onPause()
  registerReceiver(networkStateReceiver, IntentFilter(ConnectivityManager.CONNECTIVITY_ACTION))

  window.setSoftInputMode(
          WindowManager.LayoutParams.SOFT_INPUT_STATE_ALWAYS_HIDDEN
  )

  if (recordingThread != null)
      chatPresenter.startHotwordDetection()

  if (etMessage.text.toString().isNotEmpty()) {
      btnSpeak.setImageResource(R.drawable.ic_send_fab)
      etMessage.setText(“”)
      chatPresenter.micCheck(false)
  }

  chatPresenter.checkPreferences()
}

The above function sends the activity to the onPause() lifecycle method to put the activity in a pausing state so that all that is taking place right now in the activity stops and this definitely serves our purpose. But after pausing the activity, there was a further need to perform some functionality that had to done on the start of the activity and therefore the code for that was also added in this function.

How to catch STOP?

The below code was added in ChatPresenter.kt file in which if the actionType from the psh object of type ParseSusiResponseHelper is “STOP” then the view function stopMic() is called which was defined above.

val psh = ParseSusiResponseHelper()
psh.parseSusiResponse(susiResponse, i, utilModel.getString(R.string.error_occurred_try_again))

var setMessage = psh.answer
if (psh.actionType == Constant.ANSWER && (PrefManager.checkSpeechOutputPref() && check || PrefManager.checkSpeechAlwaysPref())) {
  setMessage = psh.answer

  var speechReply = setMessage
  if (psh.isHavingLink) {
      speechReply = setMessage.substring(0, setMessage.indexOf(“http”))
  }
  chatView?.voiceReply(speechReply, susiResponse.answers[0].actions[i].language)
} else if (psh.actionType == Constant.STOP) {
  setMessage = psh.stop
  chatView?.stopMic()
}

Final Output

References

  1. Stop  json response from susi server : https://api.susi.ai/susi/chat.json?timezoneOffset=-330&q=susi+stop
  2. Android life cycle methods – Google: https://developer.android.com/guide/components/activities/activity-lifecycle
  3. Interaction between view and presenters : https://medium.com/@cervonefrancesco/model-view-presenter-android-guidelines-94970b430ddf
Continue Reading“STOP” action in SUSI Android App

Add auto copy code feature in botbuilder

SUSI botbuilder lets you create your own skill bot and deploy on your website. After you have customised your bot, you will get a javascript code that you need to paste in your website’s source code. To make the process of copying the code easy, we have developed a feature for auto copying of the code to your clipboard. You just need to click on a button “copy” and the code will be copied to your clipboard.

Installing package

We use react-copy-to-clipboard package to enable auto copy feature. Install it in your project using the following command.

npm i react-copy-to-clipboard --save

 

Adding code inside render function

Inside the render() function in react file, paste the following code where you want the copy button to be displayed. Here we need to provide the text to be copied to the CopyToClipboard component via the text props. We pass the script code. We get the access token for the bot via the saved cookie. Inside the children of CopyToClipboard we need to pass the copy button which we want to show.

<CopyToClipboard
  text={
    " +cookies.get('uuid') +"' data-group='" +
    group + "' data-language='" + language + "' data-skill='" + skill + "' src='" + api + "/susi-chatbot.js'>"
  }
  onCopy={() => this.setState({ copied: true })}
>
  <span className="copy-button">copy</span>
</CopyToClipboard>

 

Thus, when the user clicks on the “copy” button, the code will be automatically copied to the user’s clipboard.

Showing snackbar message

After the code has been copied to the user’s clipboard, we can show a snackbar message to inform the user. First we pass a function onCopy to the CopyToClipboard component. This sets the state variable copied to true. Then we have a snackbar component which displays the message.

<Snackbar
   open={this.state.copied}
   message="Copied to clipboard!"
   autoHideDuration={2000}
   onRequestClose={() => {
      this.setState({ copied: false });
   }}
/>

 

Result:

 

Resources

 

Continue ReadingAdd auto copy code feature in botbuilder

Create custom theme for SUSI chatbot web plugin

SUSI web plugin bot provides the features of SUSI as a chat window which users can plugin to their websites. They just need to copy the generated javascript code into their website and the SUSI bot plugin will be enabled. The user can customize the theme of the chatbot. This blog explains how the feature of applying custom theme is implemented.

Accepting user’s input

The react component of our concern is BotBuilderPages/Design.js. We use material-ui-color-picker component to accept user’s choice of color.

<ColorPicker
  className='color-picker'
  style={{display:'inline-block',float:'left'}}
  name='color'
  defaultValue={ this.state[component.component] }
  onChange={(color)=>
   this.handleChangeColor(component.component,color) }
/>

 

Similarly, there is a text field which accepts the url of an image. The values are stored in the component’s state variables.

Storing settings in server

The design settings are stored in the text format:

This forms part of the skill’s text file. When the user clicks on “save and deploy” button, the complete skill gets send to the server through the following API:

let settings = {
      async: true,
      crossDomain: true,
      url:
        urls.API_URL +
        '/cms/' +
        (this.state.updateSkillNow ? 'modifySkill.json' : 'createSkill.json'),
      method: 'POST',
      processData: false,
      contentType: false,
      mimeType: 'multipart/form-data',
      data: form,
    };

    $.ajax(settings)
      .done(function(response) {
        // successfully saved in server
      });

 

In the server, the skill is saved in susi_private_skill_data directory. Also, the design and configuration settings are stored in chatbot.json file.

Applying the settings to the bot

Now, in the susi-chatbot.js file, the custom theme settings are applied to the bot. The function getTheme() fetches the theme settings from the server via an ajax request. Then the function applyTheme() is executed which applies the theme to the chatbot.

$(".susi-sheet-content-container").css("background-color",botbuilderBackgroundBody);
if(botbuilderBodyBackgroundImg){
$(".susi-sheet-content-container").css("background-image","url("+botbuilderBodyBackgroundImg+")");
}

 

Similarly, other theme variables are applied as well. Thus we have customised the theme of the SUSI chatbot plugin.

Result:

Resources

 

Continue ReadingCreate custom theme for SUSI chatbot web plugin

Adding typing animation and messages in SUSI Web bot plugin

SUSI web plugin bot provides the features of SUSI as a chat window which users can plugin to their websites. They just need to copy the generated javascript code into their website and the SUSI bot plugin will be enabled. This blog explains about the process of creating messages from both the user side and the bot side, and adding typing animation in the SUSI web chat plugin. A live demo of the bot can be found at susi-chatbotplugin-demo.surge.sh.

Creating User’s message

The main javascript file of our concern is skills.susi.ai/public/susi-chatbot.js. When the user types their message and press enter, setUserResponse() function is executed:

This function adds a message box to the chat window. The message box contains the message of the user.
Result:

Adding typing animation

After the user types the message and the message box is displayed, setUserResponse() function is executed. This function sets up a message box from the bot’s side and fills it with a loading gif. The important thing to note is msgNumber variable. For each user message, this variable is incremented by one. So it keeps count of the total number of message from the user or the bot. Each message box from the bot is assigned a unique id: “susiMsg-<msgNumber>”. Thus, when the response from the SUSI server is received, the loading gif is replaced by the message from the server. The corresponding message box is identified by the above id.

This function adds a message box to the chat window containing the loading gif.
Result:

On receiving the response from the server, the following function is executed:

function setBotResponse(val,msgNumber) {
    val = val.replace(new RegExp('\r?\n','g'), '<br />');
    $("#susiMsg-"+msgNumber+" .susi-msg-content-div").text(val);
    scrollToBottomOfResults();
}

 

This function replaces the above loading gif with the server’s response.

Final result:

Resources

Continue ReadingAdding typing animation and messages in SUSI Web bot plugin

Deploying SUSI Zulip bot

Zulip is a popular Real time messaging system, which combines the immediacy of Slack with an email threading model. The SUSI Zulipbot is a custom chatbot for zulip platform which fetches the response from the SUSI Server and will have some additional zulip platform specific features too. Users can install the bot into their zulip workspaces and then interact with the bot. They can either talk to the bot in private message or talk in group channels. This blog walks through the process of deploying SUSI Zulip bot into your workspace.

Cloning python-zulip-api

Python-zulip-api is where all the bots being developed for the Zulip platform can be found. The SUSI bot can be found in zulip_bots/bots/susi. susi.py is the main file where the bot’s code resides. test_susi.py is the file where the test cases for the bot are written. To clone and run the bot locally, make sure you have python3, pip and virtualenv installed. Then follow the below steps:

  1. git clone https://github.com/zulip/python-zulip-api.git  – clone the python-zulip-api repository.
  2. cd python-zulip-api  – navigate into your cloned repository.
  3. python3 ./tools/provision  – install all requirements in a Python virtualenv.
  4. The output of provision  will end with a command of the form source …/activate; run that command to enter the new virtualenv.
  5. Finished. You should now see the name of your venv preceding your prompt, e.g. (zulip-api-py3-venv)

For more information about installing the repository, refer https://zulipchat.com/api/writing-bots

Testing the bot’s output locally

For quick testing of your bot’s output, zulip-terminal is a very useful tool. It provides you a testing environment inside your terminal. After installing the above requirements, run  zulip-terminal susi in your terminal to enable testing the bot’s output:

Enter your message: hi
Reply from the bot is printed between the dotted lines:
——-
Hello!
——-
Enter your message: tell me a joke
Reply from the bot is printed between the dotted lines:
——-
It is said that looking into Chuck Norris’ eyes will reveal your future. Unfortunately, everybody’s future is always the same: death by a roundhouse-kick to the face.
——-
Enter your message: who created you?
Reply from the bot is printed between the dotted lines:
——-
The FOSSASIA community created me
——-
Enter your message: ^C
Ok, if you’re happy with your terminal-based testing, try it out with a Zulip server.
You can refer to https://zulipchat.com/api/running-bots#running-a-bot.

Deploying the bot in a Zulip workspace

Now you can deploy the bot in your own workspace or even in chat.zulip.org workspace. Follow these steps:

  1. After logging to your workspace, go to Settings () -> Your bots -> Add a new bot. Select Generic bot for bot type, fill out the form and click on Create bot.
  2. A new bot user should appear in the Active bots panel.
  3. Download the bot’s zuliprc configuration file to your computer.
  4. Run  zulip-run-bot susi –config-file ~/zuliprc-my-bot (using the path to the zuliprc file from above step).
  5. Congrats! Your bot should be running. Talk to your bot with mentioning @susi in a group channel or directly in a private channel.

Resources

Continue ReadingDeploying SUSI Zulip bot

How to make SUSI AI Line Bot

In order to integrate SUSI’s API with Line bot you will need to have a line account first so that you can follow below procedure. You can download app from here.

Pre-requisites:

  • Line app
  • Github
  • Heroku

    Steps:
    1. Install Node.js from the link below on your computer if you haven’t installed it already https://nodejs.org/en/.
    2. Create a folder with any name and open shell and change your current directory to the new folder you created.
    3. Type npm init in command line and enter details like name, version and entry point.
    4. Create a file with the same name that you wrote in entry point in above given step. i.e index.js and it should be in same folder you created.
    5. Type following commands in command line  npm install –save @line/bot-sdk. After bot-sdk is installed type npm install –save express after express is installed type npm install –save request when all the modules are installed check your package.json modules will be included within dependencies portion.

      Your package.json file should look like this.

      {
      "name": "SUSI-Bot",
      "version": "1.0.0",
      "description": "SUSI AI LINE bot",
      "main": "index.js",
      "dependencies": {
         "@line/bot-sdk": "^1.0.0",
         "express": "^4.15.2",
         "request": "^2.81.0"
      },
      "scripts": {
         "start": "node index.js"
       }
      }
    6. Copy following code into file you created i.e index.js
      'use strict';
      const line = require('@line/bot-sdk');
      const express = require('express');
      var request = require("request");
      
      // create LINE SDK config from env variables
      
      const config = {
         channelAccessToken: process.env.CHANNEL_ACCESS_TOKEN,
         channelSecret: process.env.CHANNEL_SECRET,
      };
      
      // create LINE SDK client
      
      const client = new line.Client(config);
      
      
      // create Express app
      // about Express: https://expressjs.com/
      
      const app = express();
      
      // register a webhook handler with middleware
      
      app.post('/webhook', line.middleware(config), (req, res) => {
         Promise
             .all(req.body.events.map(handleEvent))
             .then((result) => res.json(result));
      });
      
      // event handler
      
      function handleEvent(event) {
         if (event.type !== 'message' || event.message.type !== 'text') {
             // ignore non-text-message event
             return Promise.resolve(null);
         }
      
         var options1 = {
             method: 'GET',
             url: 'http://api.asksusi.com/susi/chat.json',
             qs: {
                 timezoneOffset: '-330',
                 q: event.message.text
             }
         };
      
         request(options, function(error, response, body) {
             if (error) throw new Error(error);
             // answer fetched from susi
             //console.log(body);
             var ans = (JSON.parse(body)).answers[0].actions[0].expression;
             // create a echoing text message
             const answer = {
                 type: 'text',
                 text: ans
             };
      
             // use reply API
      
             return client.replyMessage(event.replyToken, answer);
         })
      }
      
      // listen on port
      
      const port = process.env.PORT || 3000;
      app.listen(port, () => {
         console.log(`listening on ${port}`);
      });
    7. Now we have to get channel access token and channel secret to get that follow below steps.

    8. If you have Line account then move to next step else sign up for an account and make one.
    9. Create Line account on  Line Business Center with messaging API and follow these steps:
    10. In the Line Business Center, select Messaging API under the Service category at the top of the page.
    11. Select start using messaging API, enter required information and confirm it.
    12. Click LINE@ Manager option, In settings go to bot settings and Enable messaging API
    13. Now we have to configure settings. Allow messages using webhook and select allow for “Use Webhooks”.
    14. Go to Accounts option at top of page and open LINE Developers.
    15. To get Channel access token for accessing API, click ISSUE for the “Channel access token” item.
    16. Click EDIT and set a webhook URL for your Channel. To get webhook url deploy your bot to heroku and see below steps.
    17. Before deploying we have to make a github repository for chatbot to make github repository follow these steps:

      In command line change current directory to folder we created above and  write

      git init
      git add .
      git commit -m”initial”
      git remote add origin <URL for remote repository> 
      git remote -v
      git push -u origin master 

      You will get URL for remote repository by making repository on your github and copying this link of your repository.

    18. To deploy your bot to heroku you need an account on Heroku and after making an account make an app.
    19. Deploy app using github deployment method.


    20. Select Automatic deployment method.


    21. After making app copy this link and paste it in webhook url in Line channel console page from where we got channel access token.

                https://<your_heroku_app_name>.herokuapp.com/webhook
    22. Your SUSI AI Line bot is ready add this account as a friend and start chatting with SUSI.
      Here is the LINE API reference https://devdocs.line.me/en/
Continue ReadingHow to make SUSI AI Line Bot