How RSS Action Type is Implemented in SUSI Android

Important skills of SUSI.AI are to display web search queries, a map of any location and provide a list of relevant information of a topic. RSS action type is similar to websearch action type but when the web search is to be performed on the client side, it is denoted by websearch action type and when the web search is performed by the server itself, it is denoted by rss action type. In this blog, I will show you how rss action type is implemented in SUSI Android.

In case of RSS action type server searches the internet and using RSS feeds, returns an array of objects containing :

  • Title
  • Description
  • Link
{
  “title”: “dog-doh: Definitions Index”,
  “description”: “dog-doh: Definitions Index. dog dog and pony show dog biscuit dog collar dog days …”,
  “link”: “http://websters.yourdictionary.com/index/dog-doh/”,
}

title: Title related to user query

description: Description of user query.

link: If user want to know more information then user can use link to find more information.

How rss action type is parsed in SUSI Android

SUSI  reply in json format. It should be parsed properly to show it in android app. We used retrofit library developed by square to parse json data. Retrofit library parse data according to model class. We code model class according to expected json reply. For example, each susi response contains answer jsonarray. There are two jsonarray data and action inside answer jsonarray. We made a different model class for each jsonarray.

First model class is SusiResponse. We used this model class to parse ‘answers’ jsonarray.

@SerializedName(“answers”)
private List<Answer> answers = new ArrayList<>();

Here we used List<>  because ‘answers’ jsonarray contain a list of jsonarray and jsonobject. Answer is second model class. We used it to parse two important jsonarray ‘data’ and ‘action’. The ‘action’ attribute has information about action type.

public class Answer {
   @SerializedName(“data”)
   private RealmList<Datum> data = new RealmList<>();
}

Here also we used the list because data jsonarray also contains a list of jsonobject but instead of simple list we used RealmList<> because after parsing we save data using realm. ‘data’ jsonarray contain multiple jsonobject and each jsonobject contain three important information ‘title’, ‘description’ and ‘link’.

Datum class is the main model class which is used to save and retrieve ‘title’, ‘description’ and ‘link’. setTitle, setLink and setDescription method of Datum class are used to save ‘title’, ‘description’ and ‘link’ and getTitle, getDescription and getLink method are use to retrieve ‘title’, ‘description’ and ‘link’.

How rss action type data is retrieved and saved

As already mentioned we used retrofit to retrieve data and realm to save data. susiResponse is response we received from SUSI server. We used susiResponse to retrieve a list of Datum class type data.

List<Datum> datumList = susiResponse.getAnswers().get(0).getData();

We then loop through datumList and from each element we extract ‘title’, ‘description’ and ‘link’ using getTitle(), getDescription() and getLink() method respectively. Datum class is model class for both retrofit and realm. realmDatum is instance of Datum class and datumRealmList is an instance of RealmList of Datum class type. After extracting data we save data using setTitle(), setDescription() and setLink().

for (Datum datum : datumList) {
          Datum realmDatum = bgRealm.createObject(Datum.class);
          realmDatum.setDescription(datum.getDescription());
          realmDatum.setLink(datum.getLink());
          realmDatum.setTitle(datum.getTitle());
         datumRealmList.add(realmDatum);
       }      

Layout design to show rss action type reply

There are three textview with id ‘title’, ‘description’ and ‘link’ to show ‘title’, ‘description’ and ‘link’ retrieved from SUSI’s reply. We used recyclerview to show list of results.

Datum datum = datumList.get(position);

holder.titleTextView.setText(Html.fromHtml(datum.getTitle()));

holder.descriptionTextView.setText(Html.fromHtml(datum.getDescription()));

holder.descriptionTextView.setText(Html.fromHtml(datum.getDescription()));

Resources

Continue ReadingHow RSS Action Type is Implemented in SUSI Android

How Anonymous Mode is Implemented in SUSI Android

Login and signup are an important feature for some android apps like chat app because user wants to save messages and secure messages from others. In SUSI Android we save messages for logged in user on the server corresponding to their account. But  users can also  use the app without logging in. In this blog, I will show you how the anonymous mode is implemented in SUSI Android .

When the user logs in using the username and password we provide a token to user for a limited amount of time, but in case of anonymous mode we never provide a token to the user and also we set ANONYMOUS_LOGGED_IN flag true which shows that the user is using the app anonymously.

PrefManager.clearToken()

PrefManager.putBoolean(Constant.ANONYMOUS_LOGGED_IN, true)

We use ANONYMOUS_LOGGED_IN flag to check user is using the app anonymously or not. When a user opens the app we first check user is already logged in or not. If the user is not logged in then we check ANONYMOUS_LOGGED_IN flag is true or false. The true means user is using the app in anonymous mode.

if(PrefManager.getBoolean(Constant.ANONYMOUS_LOGGED_IN, false)) {

 intent = new Intent(LoginActivity.this, MainActivity.class);

startActivity(intent);

}

Else we show login page to the user. The user can use app either by login using username and password or anonymously by clicking skip button. On clicking skip button ANONYMOUS_LOGGED_IN flag set to true.

public void skip() {

Intent intent = new Intent(LoginActivity.this,MainActivity.class);

PrefManager.clearToken();

PrefManager.putBoolean(Constant.ANONYMOUS_LOGGED_IN, true);

startActivity(intent);

}

If the user is using the app in anonymous mode but he/she want to login then he/she can login. There is an option for login in menu.

When the user selects login option from the menu, then it redirects the user to the login screen and ANONYMOUS_LOGGED_IN flag is set to false. ANONYMOUS_LOGGED_IN flag is set to false to ensure that instead of login if the user closes the app and again open it, then he/she can’t use the app until logged in or click skip button.

case R.id.action_login:

PrefManager.putBoolean(Constant.ANONYMOUS_LOGGED_IN, false);

Reference

Continue ReadingHow Anonymous Mode is Implemented in SUSI Android

How Settings of SUSI Android App Are Saved on Server

The SUSI Android allows users to specify whether they want to use the action button of soft keyboard as carriage return or else send action. The user can use SUSI.AI on different client like Android , iOS, Web. To give user uniform experience, we need to save user settings on the server so that if the user makes any change in setting in any chat client then that it changes in other chat clients too. So every chat client must store user specific data on the server to make sure that all chat clients access this data and maintain the same state for that particular user and must accordingly push and pull user data from and to the server to update the state of the app.

We used special key to store setting information on server. For eg.

Setting Key Value Use
Enter As Send enter_send true/false true means pressing enter key send message and false means pressing enter key adds new line.
Mic Input mic_input true/false true means default input method is speech but supports keyboard input too. false means the only input method is keyboard input.
Speech Always speech_always true/false true means we get speech output irrespective of input type.
Speech Output speech_output true/false true means we get speech output in case of speech input and false means we don’t get speech output.
Theme theme dark/light dark means theme is dark and light means theme is light

How setting is stored to server

Whenever user settings are changed, the client updates the changed settings on the server so that the state is maintained across all chat clients. When user change setting, we send three parameters to the server ‘key’, ‘value’ and ‘token’. For eg. let ‘Enter As Send’ is set to false. When user changes it from false to true, we immediately update it on the server. Here key will be ‘enter_send’ and value will be ‘true’.

The endpoint used to add or update User Settings is :

BASE_URL+’/aaa/changeUserSettings.json?key=SETTING_NAME&value=SETTING_VALUE&access_token=ACCESS_TOKEN’

SETTING_NAME’ is the key of the corresponding setting, ‘SETTING_VALUE’ is it’s updated value and ‘ACCESS_TOKEN’ is used to find correct user account. We used the retrofit library for network call.

settingResponseCall = ClientBuilder().susiApi .changeSettingResponse(key, value,  PrefManager.getToken())

If the user successfully updated the setting on the server then server reply with message ‘You successfully changed settings of your account!’

How setting is retrieved from server

We retrieve setting from the server when user login. The endpoint used to fetch User Settings is

BASE_URL+’/aaa/listUserSettings.json?access_token=ACCESS_TOKEN’

It requires “ACCESS_TOKEN” to retrieve setting data for a particular user. When user login, we use getUserSetting method to retrieve setting data from the server. PrefManager.getToken() is used to get “ACCESS_TOKEN”.

userSettingResponseCall = ClientBuilder().susiApi .getUserSetting(PrefManager.getToken())

We use userSettingResponseCall to get a response of ‘UserSetting’ type using which we can retrieve different setting from the server. ‘UserSetting’ contain ‘Session’ and ‘Settings’ and ‘Settings’ contain the value of all settings. We save the value of all settings on the server in string format, so after retrieving settings we convert them into the required format. For eg. ‘Enter As Send’ value is of boolean format, so after retrieving we convert it to boolean format.

var settings: Settings ?= response.body().settings

utilModel.setEnterSend((settings?.enterSend.toString()).toBoolean())

Reference

Continue ReadingHow Settings of SUSI Android App Are Saved on Server

How to Show Preview of Any Link in SUSI Android App

Sometimes in a chat app like Whatsapp, messenger we find that scrolling over a link, shows a preview of the link. Android-Link-Preview is an open source library which can be used to show the preview of a link. This library is developed by Leonardo Cardoso. This library makes preview from an url using all the information such as title, relevant texts and images. TextCrawler is the main class of this library. As the name suggests this class is used to crawl given link and grab relevant information. In this blog, I will show you how I used this library in SUSI Android to show the preview of the link.

How to include this library in your project

To use android-link-preview in our project we must add the dependency of the library in build.gradle(project) file

repositories {
  maven { url https://github.com/leonardocardoso/mvn-repo/raw/master/maven-deploy’   }
}

and build.gradle(module) file.

dependencies {
   compile org.jsoup:jsoup:1.8.3 // required
   compile com.leocardz:link-preview:1.2.1@aar
}

Now import these packages in class where you want to use link preview.

import com.leocardz.link.preview.library.LinkPreviewCallback;

import com.leocardz.link.preview.library.SourceContent;

import com.leocardz.link.preview.library.TextCrawler;

As the name suggests LinkpreviewCallback is a callback which has two important methods onPre() and onPos(). onPre() method called when library starts crawling the web to find relevant information and onPos() method called when crawling finished. This library uses SourceContent class to save and retrieve data.

LinkPreviewCallback linkPreviewCallback = new LinkPreviewCallback() {

@Override

public void onPre() { }

@Override

public void onPos(final SourceContent sourceContent, boolean b) {}

}

TextCrawler class is the main class which use makePreview() method to start web crawling.

makePreview() method needs two parameters

  • linkPreviewCallback : Instance of LinkPreviewCallback class.
  • Link : Link of web page to crawl.
TextCrawler textCrawler = new TextCrawler();

textCrawler.makePreview(linkPreviewCallback, link);

So when we call makePreview method, this library starts crawling the web. But before that, it checks if given link is URL of any image or not. If given link is URL of any image, then there is no need of crawling the web because we can’t find any other information from that link, but if it is not url of any image then it starts crawling the web. All websites are written in HTML. It uses jsoup library to parse HTML documents and extract useful data from that web page. After that it store different information separately i.e it store ‘title’, ‘description’ and ‘image url’ separately using setTitle(), setDescription() and setImage() methods of SourceContent class respectively.

sourceContent.setTitle(metaTags.get(“title”));

When it finishes all these tasks, it call onPos() method of LinkPreview class to inform that it has done its task and now we can use stored data using getTitle(), getDescription() and getImage() methods inside onPos() method.

sourceContent.getTitle();

Example :

We are using Android-link-preview in our SUSI Android app

Reference

Continue ReadingHow to Show Preview of Any Link in SUSI Android App

Custom UI Implementation for Web Search and RSS actions in SUSI iOS Using Kingfisher for Image Caching

The SUSI Server is an AI powered server which is capable of responding to intelligent answers based on user’s queries. The queries to the susi server are obtained either as a websearch using the application or as an RSS feed. Two of the actions are websearch and RSS. These actions as the name suggests respond to queries based on search results from the web which are rendered in the clients. In order to use use these action types and display them in the SUSI iOS client, we need to first parse the actions looking for these action types and then creating a custom UI for them to display them.

To start with, we need to make send the query to the server to receive an intelligent response from the server. This response is parsed into different action types supported by the server and saved into relevant objects. Here, we check the action types by looping through the answers array containing the actions and based on that, we save the data for that action.

if type == ActionType.rss.rawValue {
   message.actionType = ActionType.rss.rawValue
   message.rssData = RSSAction(data: data, actionObject: action)
} else if type == ActionType.websearch.rawValue {
   message.actionType = ActionType.websearch.rawValue
   message.message = action[Client.ChatKeys.Query] as? String ?? ""
}

Here, we parsed the data response from the server and looked for the rss and websearch action type followed by which we saved the data we received from the server for each of the action types in their own objects.

Next, when a message object is created, we insert it into the dataSource item by appending it and use the `insertItems(at: [IndexPath])` method of collection view to insert them into the views at a particular index. Before adding them, we need to create a Custom UI for them. This UI will consist of a Collection View which is scrollable in the horizontal direction inside a CollectionView Cell. To start with this, we create a new class called `WebsearchCollectionView` which will be a `UIView` consisting of a `UICollectionView`.

We start by adding a collection view into the UIView inside the `init` method by overriding it. Declare a collection view using flow layout and scroll direction set to `horizontal`. Also, hide the scroll indicators and assign the delegate and datasource to `self`.

Now to populate this collection view, we need to specify the number of items that will show up. For this, we make use of the `message` variable declared. We use the `websearchData` in case of websearch action and `rssData` otherwise.

Now to specify the number of cells, we use the below method which returns the number of rss or websearch action objects and defaults to 0 such cells.

func collectionView(_ collectionView: UICollectionView, numberOfItemsInSection section: Int) -> Int {
    if let rssData = message?.rssData {
        return rssData.count
    } else if let webData = message?.websearchData {
        return webData.count
    }
    return 0
}

We display the title, description and image for each object for which we need to create a UI for the cells. Let’s start by creating a Custom Collection View cell with the imageview and 2 labels for title and description.

The imageview is given a contentMode of `aspectFit` and assigned a placeholder image in case the image doesn’t exist. The title and description labels are assigned the same font size, the title being bolder and both are center aligned.

class WebsearchCell: BaseCell {

   var imageView: UIImageView = {
       let iv = UIImageView()
       iv.contentMode = .scaleAspectFit
       iv.image = UIImage(named: "placeholder")
       return iv
   }()

   let titleLabel: UILabel = {
       let label = UILabel()
       label.textColor = .black
       label.font = UIFont.boldSystemFont(ofSize: 14)
       label.textAlignment = .center
       label.numberOfLines = 2
       label.backgroundColor = Color.grey.lighten3
       return label
   }()

   let descriptionLabel: UILabel = {
       let label = UILabel()
       label.font = UIFont.systemFont(ofSize: 14)
       label.textAlignment = .center
       return label
   }()

}

Next, we add constraints for each such view adding a title and description label adding a small margin on both sides of the cell for a cleaner UI.

addSubview(imageView)
addSubview(titleLabel)
addSubview(descriptionLabel)
descriptionLabel.numberOfLines = 5
addConstraintsWithFormat(format: "H:|-4-[v0(\(frame.width * 0.4))]-4-[v1]-4-|", views: imageView, titleLabel)
addConstraintsWithFormat(format: "|-\(frame.width * 0.4 + 8)-[v0]-4-|", views: descriptionLabel)
addConstraintsWithFormat(format: "V:|-4-[v0]-4-|", views: imageView)
addConstraintsWithFormat(format: "V:|-4-[v0(44)]-4-[v1]-4-|", views: titleLabel, descriptionLabel)

Now to use this custom cell, we first need to register it with the collection view and then we can use it easily in the `cellForItemAt` method. Since we are using Kingfisher for image caching, we use the `.kf.setImage` method to download the image from a URL and cache it as soon as it is downloaded.

collectionView.register(WebsearchCell.self, forCellWithReuseIdentifier: cellId)
 func collectionView(_ collectionView: UICollectionView, cellForItemAt indexPath: IndexPath) -> UICollectionViewCell {
       if let cell = collectionView.dequeueReusableCell(withReuseIdentifier: cellId, for: indexPath) as? WebsearchCell {
           cell.backgroundColor = .white

           if message?.actionType == ActionType.rss.rawValue {
               let feed = message?.rssData?.rssFeed[indexPath.item]
               cell.titleLabel.text = feed?.title
               cell.descriptionLabel.text = feed?.desc                cell.imageView.kf.setImage(with: URL(string: feed?.rssData?.image))
           } else if message?.actionType == ActionType.websearch.rawValue {
               let webData = message?.websearchData[indexPath.item]
               cell.titleLabel.text = webData?.title
               cell.descriptionLabel.text = webData?.desc.html2String                cell.imageView.kf.setImage(with: URL(string: feed?.webData?.image))
           }
           return cell
       } else {
           return UICollectionViewCell()
       }
   }

We check the action type and assign data based on that. Also, for the images, if they don’t exist a placeholder is added.

Since we are done with the Custom UI of the cell, we need to add it to the chat cell. For that, we add this `UIView` as a subview. For reasons of reusability, the cell was extracted into a separate one and call the `prepareForReuse()` method for reusability. This is followed by adding a subview and setting constraints for each cell and assigning the message object.

class RSSCell: ChatMessageCell {

   var message: Message? {
       didSet {
           self.addWebsearchView()
       }
   }

   lazy var websearchView: WebsearchCollectionView = {
       let view = WebsearchCollectionView()
       return view
   }()

   override func setupViews() {
       super.setupViews()
       prepareForReuse()
   }

   func addWebsearchView() {
       self.addSubview(websearchView)
       self.addConstraintsWithFormat(format: "H:|[v0]|", views: websearchView)
       self.addConstraintsWithFormat(format: "V:[v0(135)]", views: websearchView)
       websearchView.message = message
   }

}
if message.actionType == ActionType.rss.rawValue || message.actionType == ActionType.websearch.rawValue {
   if let cell = collectionView.dequeueReusableCell(withReuseIdentifier: ControllerConstants.rssCell, for: indexPath) as? RSSCell {
       cell.message = message
       return cell
   } else {
       return UICollectionViewCell()
   }
}

This is all we need to add a custom UI to a chat cell in SUSI iOS, very simple and clean.

Check the screenshot below, the app in action.

Resources:

Continue ReadingCustom UI Implementation for Web Search and RSS actions in SUSI iOS Using Kingfisher for Image Caching

Using Picasso to Show Images in SUSI Android

Important skills of SUSI.AI are to display web search queries, maps of any location and provide a list of relevant information of a topic. This blog post will cover why Glide is replaced by Picasso to show images related to these action types and how it is implemented in SUSI AndroidPicasso is a powerful image downloading and caching open source library developed by Square.

Why Glide is replaced by Picasso to show images in SUSI Android?

Previously we used Glide library to show preview in SUSI Android but we replace it because it was creating an error continuously.

java.lang.IllegalArgumentException: You cannot start a load for a destroyed activity at com.bumptech.glide.manager.RequestManagerRetriever  at Com.bumptech.glide.manager.RequestManagerRetriever.get(RequestManagerRetriever.java:102) at com.bumptech.glide.manager.RequestManagerRetriever.get(RequestManagerRetriever.java:87)at com.bumptech.glide.Glide.with(Glide.java:629)
atnorg.fossasia.susi.ai.adapters.recycleradapters.WebSearchAdapter.onBindViewHolder(WebSearchAdapter.java:74)

Reason for this error is when activity destroyed and again recreated the context used by glide is old one and  that activity already destroyed .

Glide.with(context).load(imageList.get(0))

One solution of this error is to use context.getApplicationContext()  but it is a bad idea. Another solution is to replace glide by picasso and later one is good because picasso is also a very good image downloading and caching library.

To use Picasso in your project you have to add dependency in build.gradle(Module) file.

dependencies {
  
  compile “com.squareup.picasso:picasso:2.4.0”
  
}

How Picasso is used in different actiontype

Map

“actions”: [
     {
       “type”: “map”,
       “latitude”: “1.2896698812440377”,
       “longitude”: “103.85006683126556”,
       “zoom”: “13”
     }
   ]

Link we used to retrieve image url is

http://staticmap.openstreetmap.de/statucmap.php?center=latitude,longitude& zoom=zoom&size=lengthXbreadth

Picasso will load image from this url and show image in the imageview. Here mapImage is the imageview in which map image is shown.

Picasso.with(currContext).load(mapHelper.getMapURL())
                       .into(mapImage, new  com.squareup.picasso.Callback() {    
                           @Override
                           public void onSuccess() {
                               pointer.setVisibility(View.VISIBLE);
                           }
                           @Override
                           public void onError() {
                               Log.d(“Error”, “map image can’t loaded”);
                           }
                       });

WebSearch

When we query like “Search for fog” we get ‘query’ in reply from server

“query”: “fog”

Now we use this query to retrieve image url which we used in Picasso to show images.Picasso load this image into previewImageView imageview. Image url is retrieved using  DuckDuckGo api. We are using url

https://api.duckduckgo.com/?format=json&pretty=1&q=query&ia=meanings

It gives a json response which contains image url

Picasso.with(context).load(iconUrl)
      .into(holder.previewImageView, new  com.squareup.picasso.Callback() {
                  @Override
                   public void onSuccess() {
                         Log.d(“Sucess”,“image loaded successfully”);
                   }
                   @Override
                   public void onError() {
                       holder.previewImageView.setVisibility(View.GONE);
                     }
                });     

Here also com.squareup.picasso.Callback is use to find that image is loaded successfully or not.

RSS

When we query any like “dhoni” we get ‘link’ in reply from server

“title”: “Dhoni”,

“description”: “”,
“link”: “http://de.wikipedia.org/wiki/Dhoni”

We use this link in android-link-preview library to retrieve relevant image url and then Picasso use this url to load image into imageview previewImageView.

Picasso.with(currContext).load(imageList.get(0))
      .fit().centerCrop()
      .into(previewImageView);

Reference

Continue ReadingUsing Picasso to Show Images in SUSI Android

Search Functionalities in SUSI Android App Using Android SearchView Widget

Searching is a common feature that is required in most applications. But the problem in implementing searching functionality is that there is no common way to do that. People fight over whose way is best to implement search functionality. In this blog post we’ll be looking at how search functionality works in SUSI Android App and how is it implemented. We have used Android’s SearchView widget to do that. There are many other ways to do so but this one is best suited for our requirements. Let’s see how it works.

UI Components used for Searching

1. Search icon (magnifying glass icon)

In the action bar, you can see a small icon. Clicking on the icon initiates search.

2. Edit text

An Obvious requirement is an edit test to enter search query.

3. Up and Down arrow keys

Required to search through the whole app. Simply use the up and down arrow keys to navigate through the app and find out each occurrence of the word you want to search.

 

 

 

 

 

 

 

4. Cross Button

Last but not the least, a close or cross button to close the search action.

Implementation

We have used Android’s inbuilt Widget SearchView. According to official android documentation

A widget that provides a user interface for the user to enter a search query and submit a request to a search provider. Shows a list of query suggestions or results, if available, and allows the user to pick a suggestion or result to launch into.

This widget makes searching a lot easier. It provides all methods and listeners which are actually required for searching. Let’s cover them one by one.

  1. Starting the search: searchView.setOnSearchClickListener Listener simply activates when a user clicks on search icon in the toolbar. Do all your work which needs to be done at the starting of the search like, hiding some other UI elements of doing an animation inside the listener
searchView.setOnSearchClickListener({
   chatPresenter.startSearch()
})
  1. Stop the Search: searchView.setOnCloseListener Listener gets activated when a user clicks on the cross icon to close the search. Add all the code snippet you want which is needed to be executed when the search is closed inside this like maybe notify the adapter about data set changes or closing the database etc.
searchView.setOnCloseListener({
   chatPresenter.stopSearch()
   false
})
  1.  Searching a query:  searchView.setOnQueryTextListener Listener overrides 2 methods:

3.1 onQueryTextSubmit: As the name suggests, this method is called when the query to be searched is submitted.

3.2 onQueryTextChange: This method is called when query you are writing changes.

We, basically wanted same thing to happen if user has submitted the query or if he is still typing and that is to take the query at that particular moment, find it in database and highlight it. So, chatPresenter.onSearchQuerySearched(query) this method is called in both onQueryTextSubmit and onQueryTextSubmit  to do that.

 searchView.setOnQueryTextListener(object : SearchView.OnQueryTextListener {
 
      override fun onQueryTextSubmit(query: String): Boolean {
           //Handle Search Query
           chatPresenter.onSearchQuerySearched(query)
           recyclerAdapter.query = query
           return false
       }

       override fun onQueryTextChange(newText: String): Boolean {
           if (TextUtils.isEmpty(newText)) {
               modifyMenu(false)
               recyclerAdapter.highlightMessagePosition = -1
               recyclerAdapter.notifyDataSetChanged()
               if (!editText.isFocused) {
                   editText.requestFocus()
               }
           } else {
               chatPresenter.onSearchQuerySearched(newText)
               recyclerAdapter.query = newText
           }
           return false
       }
   })
   return true
}
  1. Finding query in database: Now we have a query to be searched, we can just use a database operation to do that. The below code snippet finds all the messages which has the query present in it and work on it. If the query is not found, it simply displays a toast saying “Not found”
override fun onSearchQuerySearched(query: String) {
   chatView?.displaySearchElements(true)
   results = databaseRepository.getSearchResults(query)
   offset = 1
   if (results.size > 0) {
       chatView?.modifyMenu(true)
       chatView?.searchMovement(results[results.size - offset].id.toInt())
   } else {
       chatView?.showToast(utilModel.getString(R.string.not_found))
   }
}

This is the database operation.

override fun getSearchResults(query: String): RealmResults<ChatMessage> {
   return realm.where(ChatMessage::class.java).contains(Constant.CONTENT,
           query, Case.INSENSITIVE).findAll()
}

  1. Highlighting the part of message: Now, we know which message has the query, we just want to highlight it with a bright color to display the result. For that, we used SpannableString to highlight a part of complete string.
String text = chatTextView.getText().toString();
SpannableString modify = new SpannableString(text);
Pattern pattern = Pattern.compile(query, Pattern.CASE_INSENSITIVE);
Matcher matcher = pattern.matcher(modify);
while (matcher.find()) {
   int startIndex = matcher.start();
   int endIndex = matcher.end();
   modify.setSpan(new BackgroundColorSpan(Color.parseColor("#ffff00")), startIndex, endIndex, Spannable.SPAN_EXCLUSIVE_EXCLUSIVE);
}
chatTextView.setText(modify);

Summary

The whole point of this blog post was to educate about SearchView widget of android and how it makes it easy to search queries. All the methods you need are already implemented. You just need to call them and add database operation.

Resources

  1. The link to official android documentation explaining about different methods in SearchView Class https://developer.android.com/reference/android/widget/SearchView.html
  2. Another tutorial about SearchView http://www.journaldev.com/12478/android-searchview-example-tutorial
Continue ReadingSearch Functionalities in SUSI Android App Using Android SearchView Widget

Controlling Camera Actions Using Voice Interaction in Phimpme Android

In this blog, I will explain how I implemented Google voice actions to control camera features on the Phimpme Android project. I will cover the following features I have implemented on the Phimpme project:

  • Opening the application using Google Voice command.
  • Switching between the cameras.
  • Clicking a Picture and saving it through voice command.

Opening application when the user gives a command to Google Now.                       When the user gives command “Take a selfie” or “Click a picture” to Google Now it directly opens Phimpme camera activity.

 First                                                                                                                                        We need to add an intent filter to the manifest file so that Google Now can  detect Phimpme camera activity

<activity
   android:name=".opencamera.Camera.CameraActivity"
   android:screenOrientation="portrait"
   android:theme="@style/Theme.AppCompat.NoActionBar">
   <intent-filter>
       <action android:name="android.media.action.IMAGE_CAPTURE"/>

       <category android:name="android.intent.category.DEFAULT"/>
       <category android:name="android.intent.category.VOICE"/>
   </intent-filter>
</activity>

category android:name=”android.intent.category.VOICE” is added to the IMAGE_CAPTURE intent filter for the Google Now to detect the camera activity. For the Google Now assistance to accept the command in the camera activity we need to add the following in the STILL_IMAGE_CAMERA intent filter in the camera activity.

<intent-filter>
   <action android:name="android.media.action.STILL_IMAGE_CAMERA"/>

   <category android:name="android.intent.category.DEFAULT"/>
   <category android:name="android.intent.category.VOICE"/>
</intent-filter>

So, now when the user says “OK Google” and “Take a Picture” the camera activity on Phimpme opens.

Integrating Google Voice assistance in Camera Activity

Second,                                                                                                                               After opening the camera application the Google Assistance should ask a question.

The cameraActivity in Phimpme can be opened in two ways, such as:

  • When opened from a different application.
  • When given the command to Goole Now assistance.

We need to check whether the camera activity is prompted from Google assistance or not to activate voice command. We will check it in onResume function.

@Override
public void onResume() {
if (CameraActivity.this.isVoiceInteraction()) {
     startVoiceTrigger();
  }
} 

If is.VoiceInteraction gives “true” then voice Assistance prompts.             Assistance to ask which camera to use

Third,                                                                                                                                 After the camera activity opens the Google assistance should ask which camera to use either front or back.

To take any voice input from the user, we to store the expected commands in VoiceInteractor.PickoptionRequest. This function listens to the command by the user. We need to add synonyms for the same command.

To choose the rear camera

VoiceInteractor.PickOptionRequest.Option rear = new VoiceInteractor.PickOptionRequest.Option(getResources().getString(R.string.camera_rear), 0);
rear.addSynonym(getResources().getString(R.string.rear));
rear.addSynonym(getResources().getString(R.string.back));
rear.addSynonym(getResources().getString(R.string.normal)); 

I added synonyms like the rear, normal and back.

To choose front camera

VoiceInteractor.PickOptionRequest.Option front = new VoiceInteractor.PickOptionRequest.Option(getResources().getString(R.string.camera_front), 1);
front.addSynonym(getResources().getString(R.string.front));
front.addSynonym(getResources().getString(R.string.selfie_camera));
front.addSynonym(getResources().getString(R.string.forward));

I added synonyms like the front, selfie camera and forward. 

For assistance to ask any question such as “Which camera to use we” I have used getVoiceinteractor and inflating VoiceInteractor.PickOptionRequest.option[] array with options front and rear.

CameraActivity.this.getVoiceInteractor()
     .submitRequest(new VoiceInteractor.PickOptionRequest(
           new VoiceInteractor.Prompt(getResources().getString(“Which camera would you like to use?”),
           new VoiceInteractor.PickOptionRequest.Option[]{front, rear},
           null) {

The google assistance waits for a response from the user for only a few seconds and it goes inactive. If the user gives any unexpected command the assistance will ask the question one more time.

Check if the user gives an expected command or not.

We will override OnOptionResult(boolean finished, Options[] selection, Bundle result) function.if  (finished && selections.length == 1) if the speech length matches with any of the options provided it checks which option was used.

Check the command given by the user to switch between the cameras.

Two array objects are passed 0 and 1.  If the command given was “rear” then selection[0].getindex() = 0 and camera activity switches to the rear camera and if the the command given by the user is rear then selection[0].getIndex = 1 and camera activity switches to front camera.

@Override
public void onPickOptionResult(boolean finished, Option[] selections, Bundle result) {
  if (finished && selections.length == 1) {
     Message message = Message.obtain();
     message.obj = result;
     if (selections[0].getIndex() == 0)
     {  rearCamera();
        asktakePicture();
     }
     if (selections[0].getIndex() == 1)
     {
        asktakePicture();
     }
  }else{

       getActivity().finish();
  }

Click Picture when the user says “Cheese

After switching the camera the assistant prompts the message”Say cheese”. We need to add voiceInteractor.prompt(“Say cheese”).

We need to store the synonyms in VoiceInteractor.PickOption.Options options. I have added synonyms like ready, go, take it, OK, and Cheese to click a picture. If the user gives an unexpected output the assistance checks selection.length ==1 or not and prompts the message “Say cheese” again.

private void asktakePicture() {
  VoiceInteractor.PickOptionRequest.Option option = new VoiceInteractor.PickOptionRequest.Option(getResources().getString(R.string.cheese), 2);
  option.addSynonym(getResources().getString(R.string.ready));
  option.addSynonym(getResources().getString(R.string.go));
  option.addSynonym(getResources().getString(R.string.take));
  option.addSynonym(getResources().getString(R.string.ok));
getVoiceInteractor()
        .submitRequest(new VoiceInteractor.PickOptionRequest(
              new VoiceInteractor.Prompt(getResources().getString(R.string.say_cheese)),
              new VoiceInteractor.PickOptionRequest.Option[]{option},
              null) {
           @Override
           public void onPickOptionResult(boolean finished, Option[] selections, Bundle result) {
              if (finished && selections.length == 1) {
                 Message message = Message.obtain();
                 message.obj = result;
                 takePicture();
              } else {
                 getActivity().finish();
              }
           }
           @Override
           public void onCancel() {
              getActivity().finish();
           }
        });                                                                                                                                     

Conclusion

Now, Users can start camera activity on Phimpme through voice command “Take a Selfie”. They can switch between the cameras through voice command “Selfie camera” or “back camera”, “back” or “front” and finally click a picture by giving voice command “Cheese”, “Click it” and related synonyms.

Github

Resources

Continue ReadingControlling Camera Actions Using Voice Interaction in Phimpme Android

Showing RSS and Table Type Replies in SUSI Messenger Bots

All the messengers have a “plain text” reply support. To show web search (RSS) or table type replies, either:

  1. We need a “list type” (as in Facebook messenger) or “table type” reply support built in the messenger itself.
                                                                  or
  2. We need to convert the RSS or table type reply by SUSI API to plain text, so that we can send it, due to the “plain text” reply support available in almost every messenger.

The second point is the most favorable approach, as that way, replying with RSS or table type results is dependent only on the text support feature in the messenger. This way RSS or table type replies can be shown in messengers like REST API Gitter (which do not provide any other reply type support except text).

In SUSI web app, the UI of the web search and table type results:

As the SUSI web app is made in React js, it provided the app with necessary features to show the results this way. The messengers may not be having these required features.

So the task is we need to convert the RSS or table type replies by SUSI API to plain text to send it to the messenger.

Let’s work it out.

Converting RSS results to text:

First get familiar with the SUSI API reply to query “why” by visiting this url – http://api.susi.ai/susi/chat.json?q=why.

The JSON object returned will be constituting an array of objects as the value of the data key like:

"data": [
        {
        "title": "Why is Oracle male?",
        "description": "Why is Oracle male?. http dba oracle com why is male htm. Oracle Why is masculine?. ",
        "link": "http://dba-oracle.com/t_why_is_oracle_male.htm"
      }
]

If we notice carefully each object has 3 main keys namely “title”, “description” and “link”. So extracting these 3 properties from each object and binding them together into 1 string is the task we need to do.

So we traverse each object (i.e. rss result) in the data array and we keep on appending the values of “title”, “description” and “link” key values to the ans variable. At the end we send this resultant string to the messenger bot as a reply.  

Suppose we have the returned JSON object, in the data variable.

// storing the number of rss results
var metaCnt = data.answers[0].metadata.count;
    for(var i=0;i<metaCnt;i++){
        ans += ('Title : ');
ans += data.answers[0].data[i].title+', ';
        ans += ('Link : ');
        ans += data.answers[0].data[i].link+', ';
        ans += '\n\n';
    }

// send the message in ans variable to the messenger

Converting table type replies to text:

First get familiar with the SUSI API reply to query “why” by visiting this url – http://api.susi.ai/susi/chat.json?q=universities+in+australia.

The JSON object returned will be constituting an array of objects representing universities as the value of the data key in this form:

{
    "alpha_two_code": "AU",
    "name": "Australian Correspondence Schools",
    "country": "Australia",
    "web_page": "http://www.acs.edu.au/"
}

Here too, each object has 3 main keys namely “name”, “country” and “web_page”. So extracting these 3 properties from each object and binding them together into 1 string can make the things work.

Again traverse each object (i.e. university object) in the data array and we keep on appending the values of “name”, “country” and “web_page” key values to the ans variable. At the end we send this resultant string to the messenger bot as a reply.

Suppose we have the returned JSON object in the data variable.

    // the 3 main columns which we need are stored in colNames variable
    var colNames = data.answers[0].actions[0].columns;
    
    // storing the number of table entries
    var metaCnt = data.answers[0].metadata.count;
    for(var i=0;i<metaCnt;i++){
        for(var cN in colNames){
            // The column name
            ans += (colNames[cN]+' : ');
            // value for that column name
            ans += data.answers[0].data[i][cN]+', ';
        }
        ans += '\n\n';
    }

    // send the message in ans variable to the messenger

Resources

  1. By Slobodan Stojanović from smashing magazineDevelop a chat bot with node js.
  2. By Mikhail Larionov from Facebook blogsList templates and check box plugin
Continue ReadingShowing RSS and Table Type Replies in SUSI Messenger Bots

Using Templates in SUSI FBbot

The SUSI AI Fbbot previously showed rss and table type replies as plain text to the user. To enhance the user experience, Facebook provides with templates which can be used in it’s messenger. Using those, we show rss and table type replies wrapped up in a better U.I. creating a better user experience.

The 4 basic template structures that can be used for this task are:

  1. Button template
  2. Generic template
  3. List template
  4. Receipt template

List template is used in SUSI A.I. Fbbot because rss reply and table type reply both are lists of data.
The basic syntax for list template with reference to Fb official docs is:

"message": {
    "attachment": {
        "type": "template",
        "payload": {
            "template_type": "list",
            "top_element_style": "compact",
            "elements": [
                {
                    "title": "Classic White T-Shirt",
                    "subtitle": "100% Cotton, 200% Comfortable",
                    "default_action": {
                        "type": "web_url",
                        "url": "https://peterssendreceiveapp.ngrok.io/view?item=100"
                    },
                    "buttons": [
                        {
                            "title": "Buy",
                            "type": "web_url",
                            "url": "https://peterssendreceiveapp.ngrok.io/shop?item=100"                     
                        }
                    ]                
                }
            ]
        }
    }
}

This code shows a reply to the user like this:

If we want to show a “View more” button at the end of the list, we can add a “buttons” key and an array as its value which will have information regarding all the buttons we want to show.

The code below will show a “View more” button at the end of the list:

"elements": [
            {
                // all the elements, like shirt in the above case               
            }
       ],
       "buttons": [
            {
                "title": "View More",
                "type": "postback",
                "payload": "payload"                        
               }
       ]

Let’s learn how to incorporate these features to rss results in SUSI Fbbot:

When sending a reply using list template, “elements” key must have an array type value which will be constituted of list items. Therefore, we need to push all the rss results into that array and set it as the value of the “elements” key.

Let’s develop the code part:

Fetch the JSON object from SUSI API url with query as “why” i.e. http://api.susi.ai/susi/chat.json?q=why. Let body be a variable which stores the returned JSON object.

  • The below code fills up an array (namely elementsVal) with all the rss results:
var elementsVal = [];
var metaCnt = body.answers[0].metadata.count;
for(var i=0;i<((metaCnt>4)?4:metaCnt);i++){
    elementsVal.push(
        {
            "title": body.answers[0].data[i].title,
            "subtitle": body.answers[0].data[i].link
        }
    );
}
  • Setting the elementsVal array as the value of the “elements” key:
var message = {
    "type": "template",
    "payload": 
    {
        "template_type": "list",
        "top_element_style": "compact",
        "elements": elementsVal
    }
};
  • Sending this message as a reply to the user:
sendTextMessage(sender, message);

Same procedure can be used to render table type replies in the bot using the list template.

Generic template provided by Facebook can also used to render the web and table type replies. This template helps in showing the results in square boxes, which can be changed using left or right arrows.

Resources:

  1. By Mikhail Larionov from Facebook blogsList templates and check box plugin
  2. By Slobodan Stojanović from smashing magazineDevelop a chat bot with node js.
Continue ReadingUsing Templates in SUSI FBbot