Hotword Recognition in SUSI iOS

Hot word recognition is a feature by which a specific action can be performed each time a specific word is spoken. There is a service called Snowboy which helps us achieve this for various clients (for ex: iOS, Android, Raspberry pi, etc.). It is basically a DNN based hotword recognition toolkit.

In this blog, we will learn how to integrate the snowboy hotword detection wrapper in the SUSI iOS client. This service can be used in any open source project but for using it commercially, a commercial license needs to be obtained.

Following are the files that need to be added to the project which are provided by the service itself: snowboy-detect.h libsnowboy-detect.a and a trained model file which can be created using their online service: snowboy.kitt.ai. For the sake of this blog, we will be using the hotword “Susi”, the model file can be found here.

The way how snowboy works is that speech is recorded for a few seconds and this data is detected with an already trained model by a specific hotword, now if snowboy returns a 1 means word has been successfully detected else wasn’t.

We start with creation of a wrapper class in Objective-C which can be found wrapper and the bridging header in case this needs to be added to a Swift project. The wrapper contains methods for setting sensitivity, audio gain and running the detection using the buffer. It is a wrapper class built on top of the snowboy-detect.h header file.

Let’s initialize the service and run it. Below are the steps followed to enable hotword recognition and print out whether it successfully detected the hotword or not:

  • Create a ViewController class with extensions
    • AVAudioRecorderDelegate
    • AVAudioPlayerDelegate

since we will be recording speech.

  • Import AVFoundation
  • Create a basic layout containing a label which detects whether hotword detected or not and create corresponding `IBOutlet` in the ViewController and a button to trigger the start and stop of recognition.
  • Create the following variables:
    • let WAKE_WORD = “Susi” // hotword used
    • let RESOURCE = Bundle.main.path(forResource: “common”, ofType: “res”)
    • let MODEL = Bundle.main.path(forResource: “susi”, ofType: “umdl”) //path where the model file is stored
    • var wrapper: SnowboyWrapper! = nil // wrapper instance for running detection
    • var audioRecorder: AVAudioRecorder! // audio recorder instance
    • var audioPlayer: AVAudioPlayer!
    • var soundFileURL: URL! //stores the URL of the temp reording file
    • var timer: Timer! //timer to fire a function after an interval
    • var isStarted = false // variable to check if audio recorder already started
  • In `viewDidLoad` initialize the wrapper and set sensitivity and audio gain. Recognition best happens when sensitivity is set to `0.5` and audio gain is set to `1.0` according to the docs.
override func viewDidLoad() {
    super.viewDidLoad()
    wrapper = SnowboyWrapper(resources: RESOURCE, modelStr: MODEL)
    wrapper.setSensitivity("0.5")
    wrapper.setAudioGain(1.0)
}
  • Create an `IBAction` for the button to start recognition. This action will be used to start or stop the recording in which the action toggles based on the `isStarted` variable. When true, recording is stopped and the timer invalidated else a timer is started which calls the `startRecording` method with an interval of 4 seconds.
@IBAction func onClickBtn(_ sender: Any) {
  if (isStarted) {
    stopRecording()
    timer.invalidate()
    btn.setTitle("Start", for: .normal)
    isStarted = false
  } else {
    timer = Timer.scheduledTimer(timeInterval: 4, target: self, 
    selector: #selector(startRecording), userInfo: nil, repeats: true)
    timer.fire()
    btn.setTitle("Stop", for: .normal)
    isStarted = true
  }
}
  • Next, we add the start and stop recording methods.
    • First, a temp file is created which stores the recorded audio output
    • After which, necessary record configurations are made such as setting the sampling rate.
    • The recording is then started and the output stored in the temp file.
func startRecording() {
  do {
    let fileMgr = FileManager.default
    let dirPaths = fileMgr.urls(for: .documentDirectory, in: .userDomainMask)
    soundFileURL = dirPaths[0].appendingPathComponent("temp.wav")
    let recordSettings = [AVEncoderAudioQualityKey: 
    AVAudioQuality.high.rawValue,
    AVEncoderBitRateKey: 128000,
    AVNumberOfChannelsKey: 1,
    AVSampleRateKey: 16000.0] as [String : Any]
    let audioSession = AVAudioSession.sharedInstance()
    try audioSession.setCategory(AVAudioSessionCategoryRecord)
    try audioRecorder = AVAudioRecorder(url: soundFileURL,
settings: recordSettings as [String : AnyObject])
    audioRecorder.delegate = self
    audioRecorder.prepareToRecord()
    audioRecorder.record(forDuration: 2.0)
    instructionLabel.text = "Speak wake word: \(WAKE_WORD)"print("Started recording...")
  } catch let error {
    print("Audio session error: \(error.localizedDescription)")
  }
}
  • The stop recording method, stops the audioRecorder instance and updates the instruction label to show the same.
func stopRecording() {
  if (audioRecorder != nil && audioRecorder.isRecording) {
    audioRecorder.stop()
  }
  instructionLabel.text = "Stop"
  print("Stopped recording...")
}

The final recognition is done in the `audioRecorderDidFinishRecording` delegate method which runs the snowboy detection function which processes the audio recording in the temp file by creating a buffer and storing the audio in it and giving the wrapper this buffer as input which processes the buffer and returning a `1` is hotword was successfully detected.

func runSnowboy() {
  let file = try! AVAudioFile(forReading: soundFileURL)
  let format = AVAudioFormat(commonFormat: .pcmFormatFloat32, 
  sampleRate: 16000.0, channels: 1, interleaved: false)
  let buffer = AVAudioPCMBuffer(pcmFormat: format, frameCapacity: AVAudioFrameCount(file.length))
  try! file.read(into: buffer)
  let array = Array(UnsafeBufferPointer(start: 
  buffer.floatChannelData![0], count:Int(buffer.frameLength)))
  // print output
  let result = wrapper.runDetection(array, length: Int32(buffer.frameLength))
  print("Result: \(result)")
}

To test this out, click the start button and speak different words and you will notice that once the Hot Word is spoken, log with `result: 1` is printed out.

The snowboy hotword recognition also offers to train the personalized model with the help of Rest Apis for which the docs can be found here. The complete project implementation can be found here.

Sources:

Adding React based World Mood Tracker to loklak Apps

loklak apps is a website that hosts various apps that are built by using loklak API. It uses static pages and angular.js to make API calls and show results from users. As a part of my GSoC project, I had to introduce the World Mood Tracker app using loklak’s mood API. But since I had planned to work on React, I had to go off from the track of typical app development in loklak apps and integrate a React app in apps.loklak.org.

In this blog post, I will be discussing how I introduced a React based app to apps.loklak.org and solved the problem of country-wise visualisation of mood related data on a World map.

Setting up development environment inside apps.loklak.org

After following the steps to create a new app in apps.loklak.org, I needed to add proper tools and libraries for smooth development of the World Mood Tracker app. In this section, I’ll be explaining the basic configuration that made it possible for a React app to be functional in the angular environment.

Pre-requisites

The most obvious prerequisite for the project was Node.js. I used node v8.0.0 while development of the app. Instead of npm, I decided to go with yarn because of offline caching and Internet speed issues in India.

Webpack and Babel

To begin with, I initiated yarn in the app directory inside project and added basic dependencies –

$ yarn init
$ yarn add webpack webpack-dev-server path
$ yarn add babel-loader babel-core babel-preset-es2015 babel-preset-react --dev

 

Next, I configured webpack to set an entry point and output path for the node project in webpack.config.js

module.exports = {
    entry: './js/index.js',
    output: {
        path: path.resolve('.'),
        filename: 'index_bundle.js'
    },
    ...
};

This would signal to look for ./js/index.js as an entry point while bundling. Similarly, I configured babel for es2015 and React presets –

{
  "presets":[
    "es2015", "react"
  ]
}

 

After this, I was in a state to define loaders for module in webpack.config.js. The loaders would check for /\.js$/ and /\.jsx$/ and assign them to babel-loader (with an exclusion of node_modules).

React

After configuring the basic presets and loaders, I added React to dependencies of the project –

$ yarn add react react-dom

 

The React related files needed to be in ./js/ directory so that the webpack can bundle it. I used the file to create a simple React app –

import React from 'react';
import ReactDOM from 'react-dom';

ReactDOM.render(
    <div>World Mood Tracker</div>,
    document.getElementById('app')
);

 

After this, I was in a stage where it was possible to use this app as a part of apps.loklak.org app. But to do this, I first needed to compile these files and bundle them so that the external app can use it.

Configuring the build target for webpack

In apps.loklak.org, we need to have a file by the name of index.html in the app’s root directory. Here, we also needed to place the bundled js properly so it could be included in index.html at app’s root.

HTML Webpack Plugin

Using html-webpack-plugin, I enabled auto building of project in the app’s root directory by using the following configuration in webpack.config.js

...
const HtmlWebpackPlugin = require('html-webpack-plugin');
const HtmlWebpackPluginConfig = new HtmlWebpackPlugin({
    template: './js/index.html',
    filename: 'index.html',
    inject: 'body'
});
module.exports = {
    ...
    plugins: [HtmlWebpackPluginConfig]
};

 

This would build index.html at app’s root that would be discoverable externally.

To enable bundling of the project using simple yarn build command, the following lines were added to package.json

{
  ..
  "scripts": {
    ..
    "build": "webpack -p"
  }
}

After a simple yarn build command, we can see the bundled js and html being created at the app root.

Using datamaps for visualization

Datamaps is a JS library which allows plotting of data on map using D3 as backend. It provides a simple interface for creating visualizations and comes with a handy npm installation –

$ yarn add datamaps

Map declaration and usage as state

A map from datamaps was used a state for React component which allowed fast rendering of changes in the map as the state of React component changes –

export default class WorldMap extends React.Component {
    constructor() {
        super();
        this.state = {
            map: null
        };
    render() {
        return (<div className={styles.container}>
                <div id="map-container"></div></div>)
    }
    componentDidMount() {
        this.setState({map: new Datamap({...})});
    }
    ...
}

 

The declaration of map goes in componentDidMount method because it would not be possible to start the map until we have the div with id=”map-container” in the DOM. It was necessary to draw the map only after the component has mounted otherwise it would fail due to no id=”map-container” in the DOM.

Defining data for countries

Data for every country had two components –

data = {
    positiveScore: someValue,
    negativeScore: someValue
}

 

This data is used to generate popup for the counties –

this.setState({
    map: new Datamap({
        ...
        geographyConfig: {
            ...
            popupTemplate: function (geo, data) {
                // Configure variables pScore so that it gives “No Data” when data.positiveScore is not set (similar for negative)
                return [
                    // Use pScore and nScore to generate results here
                    // geo.properties.name would give current country name
                ].join('');
            }
        }
    })
});

The result for countries with unknown data values look something like this –

Conclusion

In this blog post, I explained about introducing a React based app in app.loklak.org’s angular based environment. I discussed the setup and bundling process of the project so it becomes available from the project’s external HTTP server.

I also discussed using datamaps as a visualisation tool for data about Tweets. The app was first introduced in pull request fossasia/apps.loklak.org#189 and was improved step by step in subsequent patches.

Resources

Using Picasso to Show Images in SUSI Android

Important skills of SUSI.AI are to display web search queries, maps of any location and provide a list of relevant information of a topic. This blog post will cover why Glide is replaced by Picasso to show images related to these action types and how it is implemented in SUSI AndroidPicasso is a powerful image downloading and caching open source library developed by Square.

Why Glide is replaced by Picasso to show images in SUSI Android?

Previously we used Glide library to show preview in SUSI Android but we replace it because it was creating an error continuously.

java.lang.IllegalArgumentException: You cannot start a load for a destroyed activity at com.bumptech.glide.manager.RequestManagerRetriever  at Com.bumptech.glide.manager.RequestManagerRetriever.get(RequestManagerRetriever.java:102) at com.bumptech.glide.manager.RequestManagerRetriever.get(RequestManagerRetriever.java:87)at com.bumptech.glide.Glide.with(Glide.java:629)
atnorg.fossasia.susi.ai.adapters.recycleradapters.WebSearchAdapter.onBindViewHolder(WebSearchAdapter.java:74)

Reason for this error is when activity destroyed and again recreated the context used by glide is old one and  that activity already destroyed .

Glide.with(context).load(imageList.get(0))

One solution of this error is to use context.getApplicationContext()  but it is a bad idea. Another solution is to replace glide by picasso and later one is good because picasso is also a very good image downloading and caching library.

To use Picasso in your project you have to add dependency in build.gradle(Module) file.

dependencies {
  
  compile “com.squareup.picasso:picasso:2.4.0”
  
}

How Picasso is used in different actiontype

Map

“actions”: [
     {
       “type”: “map”,
       “latitude”: “1.2896698812440377”,
       “longitude”: “103.85006683126556”,
       “zoom”: “13”
     }
   ]

Link we used to retrieve image url is

http://staticmap.openstreetmap.de/statucmap.php?center=latitude,longitude& zoom=zoom&size=lengthXbreadth

Picasso will load image from this url and show image in the imageview. Here mapImage is the imageview in which map image is shown.

Picasso.with(currContext).load(mapHelper.getMapURL())
                       .into(mapImage, new  com.squareup.picasso.Callback() {    
                           @Override
                           public void onSuccess() {
                               pointer.setVisibility(View.VISIBLE);
                           }
                           @Override
                           public void onError() {
                               Log.d(“Error”, “map image can’t loaded”);
                           }
                       });

WebSearch

When we query like “Search for fog” we get ‘query’ in reply from server

“query”: “fog”

Now we use this query to retrieve image url which we used in Picasso to show images.Picasso load this image into previewImageView imageview. Image url is retrieved using  DuckDuckGo api. We are using url

https://api.duckduckgo.com/?format=json&pretty=1&q=query&ia=meanings

It gives a json response which contains image url

Picasso.with(context).load(iconUrl)
      .into(holder.previewImageView, new  com.squareup.picasso.Callback() {
                  @Override
                   public void onSuccess() {
                         Log.d(“Sucess”,“image loaded successfully”);
                   }
                   @Override
                   public void onError() {
                       holder.previewImageView.setVisibility(View.GONE);
                     }
                });     

Here also com.squareup.picasso.Callback is use to find that image is loaded successfully or not.

RSS

When we query any like “dhoni” we get ‘link’ in reply from server

“title”: “Dhoni”,

“description”: “”,
“link”: “http://de.wikipedia.org/wiki/Dhoni”

We use this link in android-link-preview library to retrieve relevant image url and then Picasso use this url to load image into imageview previewImageView.

Picasso.with(currContext).load(imageList.get(0))
      .fit().centerCrop()
      .into(previewImageView);

Reference

Semantic-UI Validations for Forms in Open Event Frontend

Open Event Frontend requires forms at several places like at the time of login, for creation of events, taking the details of the user, creating discount codes for tickets etc.. Validations for these forms is a must, like in the above picture, we can see that many fields like discount code, amount etc. have been left empty, these null values when stored at backend can induce errors.

Semantic-UI makes our life easier and provides us with it’s own validations. Its form validation behavior checks data against a set of criteria or rules before passing it along to the server.

Let’s now dive deeper into Semantic validations, we’ll take discount code form as our example. The discount code form has many input fields and we have put checks at all of them, these checks are called rules here. We’ll discuss all the rules used in this form one by one

  1. Empty

Here we check if the input box with the identifier discount_amount is empty or not, if it is empty, a prompt is shown with the given message.

         identifier : ‘discount_amount’,
         rules      : [
           {
             type   : ’empty’,
             prompt : this.l10n.t(‘Please enter the discount amount’)
           }
         ]

2. Checked
Here, we validate whether the checkbox is checked or not and if it is not, show corresponding message

rules      : [
   {
     type   : ‘checked’,
     prompt : this.l10n.t(‘Please select the appropriate choices’)
   }]

3. RegExp

These checks are very important in input fields requiring passwords and codes, they specify the allowed input characters

rules      : [{
   type  : ‘regExp’,
   value : ‘^[a-zA-Z0-9_-]*$’
}]

4.Custom rules

Many a times, we require some rules which are by default not given by semantic, here we can create custom rules.

Like here, we want to check whether the user has not set max value lower than min.

$.fn.form.settings.rules.checkMaxMin = () => {
     if (this.get(‘data.minQuantity’) > this.get(‘data.maxQuantity’)) {
       return false;
     }
     return true;
   };

Here, we are creating our own custom rule checkMaxMin which returns boolean value depending upon minQuantity and maxQuantity. Now, this can be directly used as a rule

identifier : ‘min_order’,
optional   : true,
rules      : [
 {
  type   : ‘checkMaxMin’,
  prompt : this.l10n.t(‘Minimum value should not be greater than maximum’)
 }]

You can find the above code here

Additional Resources

Search Functionalities in SUSI Android App Using Android SearchView Widget

Searching is a common feature that is required in most applications. But the problem in implementing searching functionality is that there is no common way to do that. People fight over whose way is best to implement search functionality. In this blog post we’ll be looking at how search functionality works in SUSI Android App and how is it implemented. We have used Android’s SearchView widget to do that. There are many other ways to do so but this one is best suited for our requirements. Let’s see how it works.

UI Components used for Searching

1. Search icon (magnifying glass icon)

In the action bar, you can see a small icon. Clicking on the icon initiates search.

2. Edit text

An Obvious requirement is an edit test to enter search query.

3. Up and Down arrow keys

Required to search through the whole app. Simply use the up and down arrow keys to navigate through the app and find out each occurrence of the word you want to search.

 

 

 

 

 

 

 

4. Cross Button

Last but not the least, a close or cross button to close the search action.

Implementation

We have used Android’s inbuilt Widget SearchView. According to official android documentation

A widget that provides a user interface for the user to enter a search query and submit a request to a search provider. Shows a list of query suggestions or results, if available, and allows the user to pick a suggestion or result to launch into.

This widget makes searching a lot easier. It provides all methods and listeners which are actually required for searching. Let’s cover them one by one.

  1. Starting the search: searchView.setOnSearchClickListener Listener simply activates when a user clicks on search icon in the toolbar. Do all your work which needs to be done at the starting of the search like, hiding some other UI elements of doing an animation inside the listener
searchView.setOnSearchClickListener({
   chatPresenter.startSearch()
})
  1. Stop the Search: searchView.setOnCloseListener Listener gets activated when a user clicks on the cross icon to close the search. Add all the code snippet you want which is needed to be executed when the search is closed inside this like maybe notify the adapter about data set changes or closing the database etc.
searchView.setOnCloseListener({
   chatPresenter.stopSearch()
   false
})
  1.  Searching a query:  searchView.setOnQueryTextListener Listener overrides 2 methods:

3.1 onQueryTextSubmit: As the name suggests, this method is called when the query to be searched is submitted.

3.2 onQueryTextChange: This method is called when query you are writing changes.

We, basically wanted same thing to happen if user has submitted the query or if he is still typing and that is to take the query at that particular moment, find it in database and highlight it. So, chatPresenter.onSearchQuerySearched(query) this method is called in both onQueryTextSubmit and onQueryTextSubmit  to do that.

 searchView.setOnQueryTextListener(object : SearchView.OnQueryTextListener {
 
      override fun onQueryTextSubmit(query: String): Boolean {
           //Handle Search Query
           chatPresenter.onSearchQuerySearched(query)
           recyclerAdapter.query = query
           return false
       }

       override fun onQueryTextChange(newText: String): Boolean {
           if (TextUtils.isEmpty(newText)) {
               modifyMenu(false)
               recyclerAdapter.highlightMessagePosition = -1
               recyclerAdapter.notifyDataSetChanged()
               if (!editText.isFocused) {
                   editText.requestFocus()
               }
           } else {
               chatPresenter.onSearchQuerySearched(newText)
               recyclerAdapter.query = newText
           }
           return false
       }
   })
   return true
}
  1. Finding query in database: Now we have a query to be searched, we can just use a database operation to do that. The below code snippet finds all the messages which has the query present in it and work on it. If the query is not found, it simply displays a toast saying “Not found”
override fun onSearchQuerySearched(query: String) {
   chatView?.displaySearchElements(true)
   results = databaseRepository.getSearchResults(query)
   offset = 1
   if (results.size > 0) {
       chatView?.modifyMenu(true)
       chatView?.searchMovement(results[results.size - offset].id.toInt())
   } else {
       chatView?.showToast(utilModel.getString(R.string.not_found))
   }
}

This is the database operation.

override fun getSearchResults(query: String): RealmResults<ChatMessage> {
   return realm.where(ChatMessage::class.java).contains(Constant.CONTENT,
           query, Case.INSENSITIVE).findAll()
}

  1. Highlighting the part of message: Now, we know which message has the query, we just want to highlight it with a bright color to display the result. For that, we used SpannableString to highlight a part of complete string.
String text = chatTextView.getText().toString();
SpannableString modify = new SpannableString(text);
Pattern pattern = Pattern.compile(query, Pattern.CASE_INSENSITIVE);
Matcher matcher = pattern.matcher(modify);
while (matcher.find()) {
   int startIndex = matcher.start();
   int endIndex = matcher.end();
   modify.setSpan(new BackgroundColorSpan(Color.parseColor("#ffff00")), startIndex, endIndex, Spannable.SPAN_EXCLUSIVE_EXCLUSIVE);
}
chatTextView.setText(modify);

Summary

The whole point of this blog post was to educate about SearchView widget of android and how it makes it easy to search queries. All the methods you need are already implemented. You just need to call them and add database operation.

Resources

  1. The link to official android documentation explaining about different methods in SearchView Class https://developer.android.com/reference/android/widget/SearchView.html
  2. Another tutorial about SearchView http://www.journaldev.com/12478/android-searchview-example-tutorial

Handling Change of Password of SUSI.AI Account

In this blog, we will talk about a very special case, where the user changes his password to his current one only, in other words, the user enters the same password in both current password and new password. This case is now being handled by SUSI.AI server.

Considering the example of SUSI.AI Web Chat, we have following dialog when the user tries to change his/her password:

Here the user can add his/her current password and new password. When the new password meets the minimum conditions (minimum 6 characters), then the user can press CHANGE button.

We make ajax call to the server with the following endpoint:

BASE_URL+'/aaa/changepassword.json?'+
            'changepassword=' + email +
            '&password=' + this.state.passwordValue +
            '&newpassword=' + this.state.newPasswordValue +
            '&access_token='+cookies.get('loggedIn');

Here we have 4 parameters:

  • changepassword: This takes the email of the current user
  • password: This is the password of the current user, which is saved in the state named “passwordValue”
  • newpassword: This is the new password which the user enters
  • access_token: These are access tokens which are fetched from cookies. These are defined on login and are deleted on logout.

This is now handled on the server by a file named PasswordChangeService.java. Here we have to check whether the newpassword and password matches or not.

In this file, we have a function named serviceImpl with return type ServiceResponse and takes in an argument: Query post (Query is the return type). The query is not the only argument, Please read from the file from resources mentioned below for all the argument. To handle our case we just need to work with the post.

We extract the password, newpassword and email as follows:

String useremail = post.get("changepassword", null);
String password = post.get("password", null);
String newpassword = post.get("newpassword",null);

So to simply handle the case where password and newpassword matches, we define an if block in java and compare these two parameters as follows:

if(password.equals(newpassword)){
            result.put("message", "Your current password and new password matches");
            result.put("accepted", false);
            return new ServiceResponse(result);
}

Here we put the message as “Your current password and new password matches” and make the accepted flag of result JSON as false. After this, we return the ServiceResponse.

Now in our web chat client, the ajax call is as follows:

$.ajax({
                url: changePasswordEndPoint,
                dataType: 'jsonp',
                crossDomain: true,
                timeout: 3000,
                async: false,
                statusCode: {
                    422: function() {
                      let msg = 'Invalid Credentials. Please check your Email or Password.';
                      let state = this.state;
                      state.msg = msg;
                      this.setState(state);
                    }
                },
                success: function (response) {
                    let msg = response.message+'\n Please login again.';
                    let state = this.state;
                    state.msg = msg;
                    state.success = true;
                    state.msgOpen = true;
                    this.setState(state);
                }.bind(this),
                error: function(jqXHR, textStatus, errorThrown) {
                    let msg = 'Failed. Try Again';
                    if (status === 'timeout') {
                      msg = 'Please check your internet connection';
                    }
                    let state = this.state;
                    state.msg = msg;
                    state.msgOpen = true;
                    this.setState(state);
                }.bind(this)
            });

In our success method of ajax call,  we receive the JSON response in a variable named response and store this in the state in variable msg and set the state of success equal to true. We then use the state and message to handle accordingly.

Our JSON object when both password and new password are same:

So this is how clients can handle accordingly to the message received from the server instead of doing this on their own end.

Resources

Change Background of Message Section in SUSI.AI Web Chat

In SUSI.AI Web Chat we pay special attentions to the UI to make it easy to use and attract more users. Many chat apps offer users customization such as changing the colors and background. As this is very popular we decided to give the option to customize the background of message section of SUSI.AI web chat. The UI of the message section had a default gray background which could not be modified by the user. The goal was now to allow the user to customize the look of his or her own client starting with the background of the message section.

We added the settings to change the background image to Custom theme menu which occurs only when the user is logged in.The option looks like this:

User can add URL of any image of his/her choice and it will be set as the background of message section.
Now let’s take a look at the implementation of this option. We added messageBackgroundImage to our state of message section and initialised it to ‘ ’ (empty string). The TextField component looks like this:

<TextField
   name="messageImg"
   style={{display:component.component==='body'?'block':'none'}}
   ref={(input) => { this.backImage = input }}
   onChange={
     (e,value)=>
     this.handleChangeMessageBackground(value) }
    value={this.state.messageBackgroundImage}
    floatingLabelText="Message Background Image URL"
 />

OnChange method handles the input URL and this calls the function handleChangeMessageBackground. This function is having following implementation:

handleChangeMessageBackground(backImage){
    this.setState({messageBackgroundImage:backImage});
  }

It sets the messageBackgroundImage equal to the URL entered by the user. Now to change the background of message section we made a custom React style object messageBackgroundStyles:

const messageBackgroundStyles = {
        backgroundImage: `url(${this.state.messageBackgroundImage})`,
        backgroundRepeat: 'no-repeat',
        backgroundSize: '100% 100%'
    }

In the above object,

backgroundImage: sets the message section background to the image on URL entered by user

backgroundImage: Does not allows to the image to be repeated.

backgroundSize: Fills the entire message section with the background image.

Now this style was added to the component of message section :

<div className='message-section'>
     <ul
      className='message-list'
      ref={(c) => { this.messageList = c; }}
      style={messageBackgroundStyles}> // Styles added
       // Some relevant code    
      </ul>
</div>

Now we have following UI, where the user can modify the message section background with image of his or her own choice. User can add any image as background of message section by adding appropriate URL. The image will cover the entire message section without repeating itself.

Background Image source:link

This gives our user custom look according to his/her own choice.

Resources

Testing Link

http://chat.susi.ai

Background Image source of featured image : link

Sorting Photos in Phimpme Android

The Phimpme Android application features a fully fledged gallery interface with an option to switch to all photos mode, albums mode and to sort photos according to various sort actions. Sorting photos via various options helps the user to get to the desired photo immediately without having to scroll down till the end in case it is the last photo in the list generated automatically by scanning the phone for images using the Android’s mediaStore class. In this tutorial, I will be discussing how we achieved the sorting option in the Phimpme application with the help of some code snippets.

To sort the all photos list, first of all we need a list of all the photos by scanning the phone using the media scanner class via the code snippet provided below:

uri = android.provider.MediaStore.Images.Media.EXTERNAL_CONTENT_URI;
      String[] projection = {MediaStore.MediaColumns.DATA};
      cursor = activity.getContentResolver().query(uri, projection, null, null, null);

In the above code we are using a cursor to point to each photos and then we are extracting the path of the images and storing it in a list using a while loop. After we generate the list of path of all the images, we have to convert the into a list of media using the file path using the code below:

for (String path : listOfAllImages) {
          list.add(new Media(new File(path)));
      }
      return list;
  }

After generating the list of all images we can sort the photos using the Android’s collection class. In Phimpme Android we provide the option to sort photos in different categories namely:

  1. Name Sort action
  2. Date Sort action
  3. Size Sort action
  4. Numeric Sort action

As sorting is somewhat heavy task so doing this in the main thread will result in freezing UI of the application so we have to put this into an AsyncTask with a progress dialog to sort the photos. After putting the above four options in the menu options. We can define an Asynctask to load the images and in the onPreExecute method of the AsyncTask, we are displaying the progress dialog to the user to depict that the sorting process is going on as depicted in the code snippet below

AlertDialog.Builder progressDialog = new AlertDialog.Builder(LFMainActivity.this, getDialogStyle());
dialog = AlertDialogsHelper.getProgressDialog(LFMainActivity.this, progressDialog,
      getString(R.string.loading_numeric), all_photos ? getString(R.string.loading_numeric_all) : getAlbum().getName());
dialog.show();

In the doInBackgroundMethod of the AsyncTask, we are sorting the list of all photos using the Android’s collection class and using the static sort method defined in that class which takes the list of all the media files as a parameter and the MediaComparator which takes the sorting mode as the first parameter and the sorting order as the second. The sorting order decides whether to arrange the list in ascending or in descending order.

getAlbum().setDefaultSortingMode(getApplicationContext(), NUMERIC);
Collections.sort(listAll, MediaComparators.getComparator(getAlbum().settings.getSortingMode(), getAlbum().settings.getSortingOrder()));

After sorting, we have to update the data set to reflect the changes of the list in the UI. This we are doing in the onPostExecute method of the AsyncTask after dismissing the progress Dialog to avoid the window leaks in the application. You can read more about the window leaks in Android due to progressdialog here.

dialog.dismiss();
mediaAdapter.swapDataSet(listAll);

To get the full source code, you can refer the Phimpme Android repository listed in the resources below.

Resources

  1. Android developer guide to mediastore class: https://developer.android.com/reference/android/provider/MediaStore.html
  2. GitHub LeafPic repository: https://github.com/HoraApps/LeafPic
  3. Stackoverflow – Preventing window leaks in Android: https://stackoverflow.com/questions/6614692/progressdialog-how-to-prevent-leaked-window
  4. Blog – Sorting lists in Android: http://www.worldbestlearningcenter.com/tips/Android-sort-ListView.htm

Teaching Susi

In this blog post, I’ll explain how you can add a skill to SUSI.

Skills to be added in this tutorial:

  1. Ask SUSI for conversion of decimal into Binary .
  2. Ask SUSI to tell a quote.
  3. Skill development in SUSI is very interesting and easy. A comprehensive guide for skill development can be found here.

We have a Susi Skill development environment based on an Etherpad. Are you unaware what an Etherpad is? It is a blank web page where you can just put in your text and everyone can collaborate.

Here is a screenshot of what etherpad looks like:

  • open http://dream.susi.ai
  • name a dream (just pick a name for your tests in lower case letters)
  • the etherpad is filled with a welcome message, just delete the content completely

Ask SUSI for conversion of decimal into Binary

“*” here represents any decimal number number.Suppose we enter a decimal number and want susi to return it’s binary equivalent. So to make our skill, first of all we should form a general query.

Query: Convert * into binary or * in binary

After defining our query we want susi to reply with relevant answer.

For taking out the conversion from decimal to binary we can use JavaScript.

We define Javascript syntax in etherpad as follows :

!javascript:Binary for $1$ = $!$
(+$1$).toString(2);
eol

Explanation :

!javascript” allows us to print like javascript console. We can add JavaScript code by using this and do mathematical calculations to convert our decimal into binary.

“Binary for $1$ = $!$”  represents output format where $1$ stores the value in “*” in query given by user to susi. $!$ will print the result of javascript code below.

“(+$1$).toString(2);” This is a single line javascript code while converts value in “$1$” to binary, It’s output is reflected in “$!$”

“eol” represents end of line, which signifies our code for skill is finished.

In etherpad our skill would look like :

“#” – used for commenting out lines in skill

#The following code returns binary equivalent of a decimal number given by user

convert * into binary || binary of *
!javascript:Binary for $1$ = $!$
(+$1$).toString(2);
eol

Screenshot:

Ask SUSI to tell a quote

We can use external API’s for providing answer to user’s query. The external API used for telling the quote is Quotes Rest API created by They Said So. It offers a very good quotes platform and also the quotes are not repeated when several requests are made continuously.

We need a query for our skill.

Query: Tell me a quote.

Now let’s say that we use this JSON response for fetching quote data.

Our JSON object looks like:

{
 success: {
  total: 1
 },
contents: {
quotes: [
{
quote: "Let our advance worrying become advance thinking and planning.",
length: "62",
author: "Winston Churchill",
tags: [
"anxiety",
"inspire",
"planning",
"time-management"
],
category: "inspire",
date: "2017-05-31",
permalink: "https://theysaidso.com/quote/o3aMOSUwOPqUnJv9YyfYHweF/winston-churchill-let-our-advance-worrying-become-advance-thinking-and-planning",
title: "Inspiring Quote of the day",
background: "https://theysaidso.com/img/bgs/man_on_the_mountain.jpg",
id: "o3aMOSUwOPqUnJv9YyfYHweF"
}
],
copyright: "2017-19 theysaidso.com"
}
}

Now we are interested in getting value corresponding to  quotes key, which takes the path:  “path”:”$.contents.quotes“.

We make output query as follows:

!console:$quote$
     {
        "url" : "http://quotes.rest/qod.json",
        "path":"$.contents.quotes"
      }
eol

Explanation :

“!console” prints the output and this is returned by SUSI.AI.

$quote$” is the key whose value will be printed from parsed JSON.It contains our quote and this is a random quote. This is the string which will be responded by SUSI.AI

“url” generated JSON response from external API.

“path” is parsing path of JSON object which we follow for getting correct response.

Our skill will look like:

tell me a quote
!console:$quote$
     {
        "url" : "http://quotes.rest/qod.json",
        "path":"$.contents.quotes"
      }
eol

Screenshot :

Resources:

Visualising Tweet Statistics in MultiLinePlotter App for Loklak Apps

MultiLinePlotter app is now a part of Loklak apps site. This app can be used to compare aggregations of tweets containing a particular query word and visualise the data for better comparison. Recently there has been a new addition to the app. A feature for showing tweet statistics like the maximum number of tweets (along with date) containing the given query word and the average number of tweets over a period of time. Such statistics is visualised for all the query words for better comparison.

Related issue: https://github.com/fossasia/apps.loklak.org/issues/236

Obtaining Maximum number of tweets and average number of tweets

Before visualising the statistics we need to obtain them. For this we simply need to process the aggregations returned by the Loklak API. Let us start with maximum number of tweets containing the given keyword. What we actually require is what is the maximum number of tweets that were posted and contained the user given keyword and on which date the number was maximum. For this we can use a function which will iterate over all the aggregations and return the largest along with date.

$scope.getMaxTweetNumAndDate = function(aggregations) {
        var maxTweetDate = null;
        var maxTweetNum = -1;

        for (date in aggregations) {
            if (aggregations[date] > maxTweetNum) {
                maxTweetNum = aggregations[date];
                maxTweetDate = date;
            }
        }

        return {date: maxTweetDate, count: maxTweetNum};
    }

The above function maintains two variables, one for maximum number of tweets and another for date. We iterate over all the aggregations and for each aggregation we compare the number of tweets with the value stored in the maxTweetNum variable. If the current value is more than the value stored in that variable then we simply update it and keep track of the date. Finally we return an object containing both maximum number of tweets and the corresponding date.Next we need to obtain average number of tweets. We can do this by summing up all the tweet frequencies and dividing it by number of aggregations.

$scope.getAverageTweetNum = function(aggregations) {
        var avg = 0;
        var sum = 0;

        for (date in aggregations) {
            sum += aggregations[date];
        }

        return parseInt(sum / Object.keys(aggregations).length);
    }

The above function calculates average number of tweets in the way mentioned before the snippet.

Next for every tweet we need to store these values in a format which can easily be understood by morris.js. For this we use a list and store the statistics values for individual query words as objects and later pass it as a parameter to morris.

var maxStat = $scope.getMaxTweetNumAndDate(aggregations);
        var avg = $scope.getAverageTweetNum(aggregations);

        $scope.tweetStat.push({
            tweet: $scope.tweet,
            maxTweetCount: maxStat.count,
            maxTweetOn: maxStat.date,
            averageTweetsPerDay: avg,
            aggregationsLength: Object.keys(aggregations).length
        });

We maintain a list called tweetStat and the list contains objects which stores the query word and the corresponding values.

Apart from plotting these statistics, the app also displays the statistics when user clicks on an individual treat present in the search record section. For this we filter tweetStat list mentioned above and get the required object corresponding to the query word the user selected bind it to angular scope. Next we display it using HTML.

<div class="tweet-stat max-tweet">
                  <div class="stat-label"> <h4>Maximum number of tweets containing '{{modalHeading}}' :</h4></div>
                  <div class="stat-value"> <strong>{{selectedTweetStat.maxTweetCount}}</strong> tweets on
                    <strong>{{selectedTweetStat.maxTweetOn}}</strong>
                  </div>
                </div>

Finally we need to plot the statistics. For this we use a function called plotStatGraph dedicated only for plotting statistics graph. We pass the tweetStat list as a parameter to morris and configure all the other parameters.

$scope.plotStatGraph = function() {
        $scope.plotStat = new Morris.Bar({
            element: 'graph',
            data: $scope.tweetStat,
            xkey: 'tweet',
            ykeys: ['maxTweetCount', 'averageTweetsPerDay'],
            labels: ['Maximum no. of tweets : ', 'Average no. of tweets/day'],
            parseTime: false,
            hideHover: 'auto',
            resize: true,
            stacked: true,
            barSizeRatio: 0.40
        });
        $scope.graphLoading = false;
    }

But now we have two graphs. One for showing variations in aggregation and the other for showing statistics. How do we manage them? Somehow we need to show them in the same page as this is a single page app. Also we need to avoid vertical scrolling as it would degrade both UI and UX. So we need to implement a switching mechanism. The user should be able to switch between the two graph views as per their wish. How to achieve that? Well, for this we maintain a global variable which will keep track of the current plot type. If the current graph type is aggregations then we call the function to plot aggregations otherwise we call the above mentioned function to plot statistics.

$scope.plotData = function() {
        $(".plot-data").html("");
        if ($scope.currentGraphType === "aggregations") {
            $scope.plotAggregationGraph();
        } else {
            $scope.plotStatGraph();
        }
    }

Lastly we integrate this state variable (currentGraphType) with the UI so that users can easily toggle between graph views with just a click.

<div class="switch" ng-click="toggle()">
                <span ng-if="queryRecords.length !== 0" class="glyphicon glyphicon-stats"></span>
              </div>

Important resources