Using Vector Images in SUSI Android

SUSI is an artificial intelligence for interactive chat bots. For making it more user friendly and interactive we add a lot of images in the form of drawable resources in the SUSI Android App (https://github.com/fossasia/susi_android). Most of these drawables are in the form of PNGs. There are certain problems associated with the use of PNG images.

  1. PNGs cannot be scaled without losing quality. Due to which for the same PNG image we have to include separate images of varied quality. Otherwise the image will become blur.
  2. PNGs tends to take large disk space which can be easily reduced with the use of vector images.
  3. PNGs have fixed color and dimensions which cannot be changed.

Due to the above shortcomings of PNG images we decided to use vector drawable images instead of them.

Advantages associated with Vector images

  1. They can be scaled to any size without the loss in quality. Thus we need to include only a single image in the app and not of varied qualities.
  2. They are very small in size as compared to PNGs.
  3. They can be easily modified programmatically in XML file unlike PNGs.

Using Vector Images in Android Studio

Android Studio provide tools by which we can directly import vector drawables in the project. To import Vector images go to File>New>Vector Assets in studio.

From here we can choose the icon we want to include in our project and click OK. The icon will appear in the drawables directory and can be used anywhere in the projects.

Implementation in SUSI Android

In Susi Android we have used various vector images such as arrows, pointer and even the logo of the app. Here below is the logo of SUSI.

This is actually a vector image below we will see the code required to get this logo as the output.

<vector android:height="50dp" android:viewportHeight="279.37604"

  android:viewportWidth="1365.2" android:width="220dp" xmlns:android="http://schemas.android.com/apk/res/android">

<path android:fillColor="#ffffff"

      android:pathData="M127.5,7.7c-26.8,3.3 -54.2,16.8 -75.9,37.4 -11.8,11.1 -20.4,22.9 -28.1,38.4 -8.9,17.8 -12.8,32.1 -13.7,51l-0.3,6 39,0 39,0 0.3,-4c0.7,-12.1 6.8,-24.1 17.2,-34.5 8.5,-8.4 16.2,-13.4 25.9,-16.7l6.6,-2.2 81.3,-0.1 81.2,0 0,-38 0,-38 -84.7,0.1c-46.7,0.1 -86.1,0.4 -87.8,0.6z" android:strokeColor="#00000000"/>

  <path android:fillColor="#ffffff"

      android:pathData="M319.2,11.3l-4.3,4.3 0.3,103c0.4,113.2 0,105.9 6.4,118.6 10.8,21.3 35.1,41.9 56.2,47.3 8.5,2.3 99.1,2.2 107.7,0 18.7,-4.9 39.2,-20.7 51.5,-39.7 3.4,-5.1 7.1,-12.2 8.3,-15.8l2.2,-6.5 0.5,-103.3 0.5,-103.3 -4.5,-4.4 -4.6,-4.5 -31.5,0 -31.5,0 -4.7,4.8 -4.7,4.8 0,93 0,93 -3.3,3.2 -3.3,3.2 -29,0 -29,0 -2.6,-2.7 -2.7,-2.8 -0.7,-94.2 -0.7,-94.2 -4.3,-4 -4.2,-4.1 -31.9,0 -31.9,0 -4.2,4.3z" android:strokeColor="#00000000"/>

  <path android:fillColor="#ffffff"

      android:pathData="M680,7.6c-31.6,4.8 -56.1,17.3 -79,40.3 -23.2,23.3 -36.3,50.5 -38.9,80.9 -0.5,5.9 -0.7,11 -0.4,11.4 0.2,0.5 17.7,0.8 38.8,0.8l38.4,0 0.6,-4.8c3.2,-23.2 21.3,-44.1 44.7,-51.3 5.6,-1.8 10.6,-1.9 86.6,-1.9l80.7,0 -0.3,-38 -0.2,-38 -84.3,0.1c-46.3,0.1 -85.3,0.3 -86.7,0.5z" android:strokeColor="#00000000"/>

  <path android:fillColor="#ffffff"

      android:pathData="M869.1,13.4l-4.1,6.4 0,126.4 0,126.3 4.8,6.7 4.7,6.8 31.6,0 31.6,0 4.7,-7 4.6,-7 0,-125.7 0,-125.8 -4.7,-6.7 -4.8,-6.8 -32.1,0 -32.1,0 -4.2,6.4z" android:strokeColor="#00000000"/>

  <path android:fillColor="#ffffff"

      android:pathData="M222.5,152.2c-0.2,0.7 -0.9,4.2 -1.5,7.7 -3.4,19.5 -19.4,38 -40,46.4l-5.5,2.2 -83,0.5 -83,0.5 -0.3,37.8 -0.2,37.8 89.2,-0.3 89.3,-0.3 9.6,-2.7c57.7,-16.3 100.1,-67.4 102.1,-123.3l0.3,-7 -38.3,-0.3c-30.1,-0.2 -38.3,0 -38.7,1z" android:strokeColor="#00000000"/>

  <path android:fillColor="#ffffff"

      android:pathData="M774.5,152.2c-0.2,0.7 -0.9,4.1 -1.5,7.5 -3.3,19.2 -18.8,37.3 -39.4,46.2l-6.1,2.6 -83,0.5 -83,0.5 -0.3,37.7 -0.2,37.8 85.9,0c93.7,0 91.4,0.1 110.1,-5.9 26.4,-8.5 53.3,-28.4 69.8,-51.7 15.2,-21.3 25.1,-50.1 24,-69.9l-0.3,-6 -37.8,-0.3c-29.7,-0.2 -37.8,0 -38.2,1z" android:strokeColor="#00000000"/>

  <path android:fillColor="#ffffff" android:pathData="m1146.99,0 l-1.38,1.19c-0.76,0.66 -1.85,1.61 -2.43,2.13 -0.58,0.51 -1.75,1.54 -2.61,2.28 -1.52,1.31 -1.58,1.41 -2.4,3.53 -0.46,1.2 -0.92,2.37 -1.01,2.59 -30.55,82.93 -61.62,165.72 -96.03,259.63 0,0.08 1.61,1.88 3.57,3.98l3.57,3.84 33.47,-0.04 33.47,-0.04c12.28,-35.6 25.13,-72.47 37.4,-107.27 0.06,-0.25 0.28,-0.64 0.5,-0.88 0.37,-0.41 0.61,-0.43 4.2,-0.43 3.63,0 3.83,0.02"/>

  <path android:fillColor="#ffffff" android:pathData="m967.09,279.18c-2.48,-3.74 -4.97,-7.04 -8.09,-11.76l0.09,-43.92c3.34,-5.26 5.31,-6.73 8.42,-11.51 17.91,0.02 34.3,0.26 50.88,0.26 3.21,4.88 4.09,6.72 7.81,12.66 -0.05,13.98 0.1,27.96 -0.12,41.94 -2.9,4.2 -4.27,7.42 -7.78,12.18 -18.81,-0.04 -35.43,0.2 -51.21,0.15z"/>

  <path android:fillColor="#ffffff"

      android:pathData="m1287.3,6.59 l-4.1,6.4 0,126.4 0,126.3 4.8,6.7 4.7,6.8 31.6,0 31.6,0 4.7,-7 4.6,-7 0,-125.7 0,-125.8 -4.7,-6.7 -4.8,-6.8 -32.1,0 -32.1,0 -4.2,6.4z" android:strokeColor="#00000000"/>


</vector>

In this code we can easily change the color and minor details for the logo which could have been not possible if the logo was in PNG format. Also we don’t need multiple logo images of varied qualities as it can be scaled without decreasing quality.

Resources

Using Day Night Theme in SUSI Android

SUSI is an artificial intelligence for interactive chat bots. It provides response to the user in most intuitive way. Therefore we thought why not implement the option to give theme preference to the user to make it more interactive. It will also help in increasing the user’s interest towards the application.

We tried out different themes and then finally decided to settle for the newly announced Day Night Theme for the SUSI Android App (https://github.com/fossasia/susi_android). This theme is provided by AppCompat 23.2.0 . With the help of this theme we can switch between Theme.AppCompat.Light (light) and Theme.AppCompat (dark) based on the user preference and time of day. For default the theme is set to the light theme and it can be easily changed from the settings. Thus it allows the user to change the theme according to his or her mood which looks very intuitive.

How to use this theme?

To use the Day Night theme is quite simple. We just need to extend our default theme to that of Theme.AppCompat.DayNight. The declaration is done as shown below in the screenshot.

<style name="MyTheme" parent="Theme.AppCompat.DayNight">

  <!-- Blah blah -->

</style>

Now to enable different features of the theme in our application we need to call AppCompatDelegate.setDefaultNightMode(). It takes one of the following values as the parameter.

  • MODE_NIGHT_NO. This is for the day (light) theme.
  • MODE_NIGHT_YES.This is for the night (dark) theme.
  • MODE_NIGHT_AUTO. It automatically changes between the above two themes based on the time of day.
  • MODE_NIGHT_FOLLOW_SYSTEM (default). This theme is dependent on the system settings of the user mobile phone.

We can set one of these parameters at the time of calling the function to fix the theme of the application in the following way.

static {

AppCompatDelegate.setDefaultNightMode(

          AppCompatDelegate.MODE_NIGHT_...);

}

The theme inside an activity is set at the time time of calling onCreate() method. Therefore we cannot change the theme from any other place inside our activity apart from onCreate(). If we want to set it inside our activity but outside the onCreate() method then we have to call the recreate() function to recreate the whole activity which will implement the selected theme.Let us look at the example.

public class MyActivity extends AppCompatActivity {

 public void onCreate(Bundle savedInstanceState) {

      super.onCreate(savedInstanceState);

      if (savedInstanceState == null) {

          // Set the local night mode to some value

          getDelegate().setLocalNightMode(

                  AppCompatDelegate.MODE_NIGHT_...);

          // Now recreate for it to take effect

          recreate();

      }

  }

}

To take care of the text colors in our app we can set textColor attribute as

?android:attr/textColorPrimary

Now let us look at the implementation in Susi Android

In Susi Android we are providing user the option to select either the dark or the light theme in the settings.

 
The code for the implementation is as below

@Override

protected void onCreate(Bundle savedInstanceState) {

  super.onCreate(savedInstanceState);



  prefs = getSharedPreferences(Constant.THEME, MODE_PRIVATE);

  if(prefs.getString(Constant.THEME,"Dark").equals("Dark")) {

      AppCompatDelegate.setDefaultNightMode(AppCompatDelegate.MODE_NIGHT_YES);

  }

  else {

      AppCompatDelegate.setDefaultNightMode(AppCompatDelegate.MODE_NIGHT_NO);

  }



  setContentView(R.layout.activity_main);

}

The result output for the light theme is

To learn more about themes in Android you can refer to this link.

Resources

Using Picasso library in SUSI Android

SUSI is an artificial intelligence for chatbots which have the ability to reply in most intuitive way through different types of answers such as images, charts, maps and text. Hence for the image displays in the SUSI Android client we need an image loading library which can help us to cache the images in the app. There are a few options available which include Glide and Picasso. Both of these libraries are open sourced. After some research we finally came up to use Picasso as it provides more additional features in comparison to Glide.

Picasso is an image downloading library. It is an open source library. It is published and maintained by Square. It allows the developer to display an image from the external URL of the image. It provides caching of image in just a few lines of code. Previously without this library it was very difficult to download and display the image and required a lot more lines of code. But with the help of Picasso this task is reduced to just a few lines of code.

How to use Picasso?

To use Picasso in our project we must add the dependency of the library in build.gradle file.

dependencies {
  ...
  compile "com.squareup.picasso:picasso:2.4.0"
  ...
}

Let us define an imageView in which we want to load the image with the help of Picasso Library.

<ImageView
   android:layout_width="wrap_content"
   android:layout_height="wrap_content"
   android:id="@+id/imageView"
   android:layout_alignParentTop="true"
   android:layout_centerHorizontal="true">
</ImageView>


Now we are all set to download the image with the help of Picasso library in the following way:-

//Initialize ImageView
ImageView imageView = (ImageView) findViewById(R.id.imageView);

//Loading image from below url into imageView

Picasso.with(this)
  .load("IMAGE URL")
  .into(imageView);

Picasso also provide the function for setting placeholders and error images to be shown if there is any problem in the downloading of the image.

Picasso.with(this)
   .load("YOUR IMAGE URL HERE")
   .placeholder(R.drawable.ic_placeholder) 
   .error(R.drawable.ic_error_fallback)         
   .into(imageView);

Now let us see the implementation of Picasso in Susi Android

In the Susi app we are storing the link of images coming from response in the imageList.

if (imageList == null || imageList.size() == 0) {

  holder.linkPreviewImageView.setVisibility(View.GONE);

} else {

  Picasso.with(context).load(imageList.get(0))

          .fit().centerCrop()

          .into(holder.linkPreviewImageView);
  }

Here we are passing the activity context to the Picasso library. We can use additional features like fit() and centerCrop() method the way we are using in the Susi app. These methods are fit the image at the center of the imageView.

Screenshots from the Susi App

Picasso Library also provides some additional functions as well like:-

Picasso.with(this)
    .load("YOUR IMAGE URL HERE")        
    .placeholder(R.drawable.ic_placeholder)   // optional        
    .error(R.drawable.ic_error_fallback)      // optional        
    .resize(250, 200)                        // optional        
    .rotate(90)                             // optional        
    .into(imageView);

You can find more about Picasso from this link.

Implementing DuckDuckGo Api in SUSI Android

As we know that Susi is an open source intelligent chatbot, it must be able to reply with user’s query on any topic. Therefore we have implemented DuckDuckGo API in Susi Android(https://github.com/fossasia/susi_android) which will help us to generate search result for the query asked by the user.

 

DuckDuckGo is an API which provides instant search results. This basically works as a search engine having information about various things. The most important thing about DuckDuckGo is that it is non tracking. It does not track its users and show results based on their search history. Thus the search results remain uniform across all the clients irrespective of their search history. The information inside the API is fed from more than 120 different independent sources. This is what makes it different from other search engines. The response in the form of answers include different types of links, description, categories, and definition about various stuffs.

For more details about the Api please check this link.

 

API endpoints:

http://api.duckduckgo.com/?q=DuckDuckGo&format=json

This is one of the links generated to test the api. Here we can see different parameters, the parameter q is the query parameter where we write our question/query to get the response from the API. The format here defines the format in which we want the response to be. Here in this case we are obtaining the response in the form of JSON which can be parsed to obtain the desired result in the client.

The response obtained by the following query is as follow:-

{  

  "DefinitionSource":"",

  "Heading":"DuckDuckGo",

  "ImageWidth":340,

  "RelatedTopics":[  

     {  

        "Result":"<a href=\"https://duckduckgo.com/Names_Database\">Names Database</a> - The Names Database is a partially defunct social network, owned and operated by Classmates.com, a wholly owned subsidiary of United Online. The site does not appear to be significantly updated since 2008, and has many broken links and display issues.",

        "Icon":{  

           "URL":"",

           "Height":"",

           "Width":""

        },

        "FirstURL":"https://duckduckgo.com/Names_Database",

        "Text":"Names Database - The Names Database is a partially defunct social network, owned and operated by Classmates.com, a wholly owned subsidiary of United Online. The site does not appear to be significantly updated since 2008, and has many broken links and display issues."

     }

}

 

Implementation in Susi android

In Susi Android we are using Retrofit library by Square for API calling. Retrofit is one of the best libraries present for the network calling. It helps the developer to migrate from the old way of using AsyncTask in the Android app which creates a lot of mess and ugly code.

For the implementation in Susi Android, we have made a WebSearchClient class that stores the base address for the API calling.

public class WebSearchClient {

  public static final String BASE_URL = "http://api.duckduckgo.com";

  private static Retrofit retrofit = null;



  public static Retrofit getClient() {

      if (retrofit==null) {

          retrofit = new Retrofit.Builder()

                  .baseUrl(BASE_URL)

                  .addConverterFactory(GsonConverterFactory.create())

                  .build();

      }

      return retrofit;

  }

}

To get the response we call the API with get method passing the query and format parameter in the following way.





public interface WebSearchService {



[email protected]("/?format=json&pretty=1")

  Call<WebSearch> getresult(@Query("q") String query);

}



The results obtained from the server in the form of JSON is parsed in the form of objects in this way:-



public class WebSearch{



[email protected]("Heading")

[email protected]

  private String heading;



[email protected]("RelatedTopics")

[email protected]

  private List<RelatedTopics> relatedTopics;



  public WebSearch(String heading, List<RelatedTopics> relatedTopics) {

      this.heading = heading;

      this.relatedTopics = relatedTopics;

  }





public class RelatedTopics {



[email protected]("FirstURL")

[email protected]

  private String url;



[email protected]("Text")

[email protected]

  private String text;



[email protected]("Icon")

[email protected]

  private WebIcon icon;



[email protected]("Result")

[email protected]

  private String result;



  public RelatedTopics(String url, String text, WebIcon icon) {

      this.url = url;

      this.text = text;

      this.icon = icon;

  }



At the end after setting up all the above things, we call the API and store the result in the realm or local storage.

final WebSearchService apiService = WebSearchClient.getClient().create(WebSearchService.class);



Call<WebSearch> call = apiService.getresult(webquery);



call.enqueue(new Callback<WebSearch>() {

[email protected]

  public void onResponse(Call<WebSearch> call, Response<WebSearch> response) {

      Log.e(TAG, response.toString());

      if (response.body() != null ) {

          realm.beginTransaction();

          RealmList<WebSearchModel> searchResults = new RealmList<>();


    }
}

This completes the calling of DuckDuckGo API and storing the results in the database.

To know more about API calling using Retrofit you can refer to this link.

Custom Views in Susi Android App

Android provides us with the ability to have different views for your App. These views help in the formation of the UI element of the application. These includes imageView, textView and layouts such as LinearLayout and FrameLayout etc. The view hierarchy of Android looks something like this.

The problem with these views is that we cannot modify them according to our own need inside the application. This is what we faced during the making of chat bubble layout for Susi Android App (https://github.com/fossasia/susi_android). We wanted to implement the chat bubble similar to Whatsapp that resizes and position the time textView according to size of the response coming from the server ie something like this as shown in the screenshot. Therefore we finally came up the solution of using Custom views inside the app that allowed us to modify the view the way we wanted.

 

So now lets us understand how we can make custom views by extending existing views

So first question that comes in our mind is why are we extending existing views if we want to make our own. The reason behind this is that extending an existing view provides us with ability to have all the features that are there in an existing view.

On top of that we can add our own functionality into it. Now see below how we can implement it.

It’s time for some actual code.

As we can see in the code below that here we made our own custom class called ValueSelector. This class is extending the existing layout which is RelativeLayout. The first constructor used in the above class which takes context as the parameter is used to create an instance of the view programmatically. The second constructor used which takes context and AttributeSet as parameters is used to inflate the view from the XML.

While the third constructor is used to define the base classes.

public class ValueSelector extends RelativeLayout {

   View rootView;
   TextView valueTextView;
   View minusButton;
   View plusButton;

   public ValueSelector(Context context) {
       super(context);
       init(context);
   }

   public ValueSelector(Context context, AttributeSet attrs) {
       super(context, attrs);
       init(context);
   }

   private void init(Context context) {
       //do setup work here
   }

The init method used here is to inflate the views and to get the reference of all the child view.

private void init(Context context) {
   rootView = inflate(context, R.layout.value_selector, this);
   valueTextView = (TextView) rootView.findViewById(R.id.valueTextView);

   minusButton = rootView.findViewById(R.id.minusButton);
   plusButton = rootView.findViewById(R.id.plusButton);

   minusButton.setOnClickListener(new View.OnClickListener() {
[email protected]
       public void onClick(View v) {
           decrementValue(); //we'll define this method later
       }
   });

   plusButton.setOnClickListener(new View.OnClickListener() {
[email protected]
       public void onClick(View v) {
           incrementValue(); //we'll define this method later        }
   });
}

Let’s now see the implementation of CustomViews in Susi Android.

 

public class ChatBubbleLayout extends FrameLayout {



  public ChatBubbleLayout(Context context) {

      super(context);

  }



  public ChatBubbleLayout(Context context, AttributeSet attrs) {

      super(context, attrs);

  }



  public ChatBubbleLayout(Context context, AttributeSet attrs, int defStyleAttr) {

      super(context, attrs, defStyleAttr);

  }



[email protected](21)

  public ChatBubbleLayout(Context context, AttributeSet attrs, int defStyleAttr, int defStyleRes) {

      super(context, attrs, defStyleAttr, defStyleRes);

  }



[email protected]

  protected void onMeasure(int widthMeasureSpec, int heightMeasureSpec) {

      super.onMeasure(widthMeasureSpec, heightMeasureSpec);

  }

 

In Susi Android app we are making our own custom view for the chat bubble layout, the view is extending the FrameLayout. Here we are overriding onMeasure method. This method is used for defining the constraint of the layout relative to that of the parent view. With the help of this we can set how big or small our layout will be.

Now lets us see the design of the custom view in Susi app.

 

      You can learn more about the custom views from this link.

Setup Lint Testing in SUSI Android

As developers tend to make mistakes while writing code, small mistakes or issues can cause a negative impact on the overall functionality and speed of the app. Therefore it is necessary to understand the importance of Lint testing in Android.

Android Lint is a tool present in Android studio which is effective in scanning and reporting different types of bugs present in the code, it also find typos and security issues in the app. The issue is reported along with severity level thus allowing the developer to fix it based on the priority and level of damage they can cause. It is easy to use and can significantly improve the quality of your code.

Effect of Lint testing on Speed of the Android App

Lint testing can significantly improve the speed of the app in the following ways

  1. Android Link test helps in removing the declaration redundancy in the code, thus the Gradle need not to bind the same object again and again helping to improve the speed.
  2. Lint test helps to find bugs related to Class Structure in different Activities of the Application which is necessary to avoid the case of memory leaks.
  3. Lint testing also tells the developer above the size of resources use for example the drawable resources which sometimes take up a large piece of memory in the application. Cleaning these resources or replacing them with lightweight drawables helps in increasing the speed of the app.
  4. Overall Lint Testing helps in removing Typos, unused import statement, redundant strings, hence refactoring the whole code and increasing stability and speed.

Setup

We can use Gradle to invoke the list test by the following commands in the root directory of the folder.

To set up we can add this code to our build.gradle file

 lintOptions {
        lintConfig file("lint.xml")
    }

The lint.xml generated will look something like this

<?xml version="1.0" encoding="UTF-8"?>
<lint>
   <!-- Changes the severity of these to "error" for getting to a warning-free build -->
   <issue id="UnusedResources" severity="error"/>
</lint>

To explicitly run the test on Android we can use the following commands.

On Windows

gradlew lint

On Mac

./gradlew lint

We can also use the lint testing on various variants of the app, using commands such as

gradle lintDebug 

  or 

gradle lintRelease.

The xml file generated contains the error along with their severity level .

<?xml version="1.0" encoding="UTF-8"?>
<lint>
   <issue id="Invalid Package" severity="ignore" />
   <!-- All below are issues that have been brought to informational (so they are visible, but don't break the build) -->
   <issue id="GradleDependency" severity="informational" />    <issue id="Old TargetApi" severity="informational" />
</lint>

Testing on Susi Android:-

After testing the result on Susi Android we find the following errors.

As we can see that there are two errors and a lot of warnings. Though warning are not that severe but we can definitely improve on this. Thus making a habit of testing your code with lint test will improve the performance of your app and will make it more faster.

The test provides a complete and detailed list of issues present in the project.

We can find the exact location as well as the cause of the error by going deeper into the directory like this.

We can see there is a error in build.gradle file which is due to different versions of libraries present in the gradle files as of com.android.support libraries must be of same version.

In this way we can test out code and find errors in it.

Using Speech To Text Engine in Susi Android

Susi is an intelligent chatbot, it supports speech to text as the input. The user can talk to the susi just like he or she is talking to some other person. Also in case of speech to text input, the output of susi is in the form of text to speech giving a seamless conversational experience to the user.

To achieve speech to text input in Susi Android or any other android application we have the following ways:-

  1. Using Android inbuilt Speech to Text function.
  2. Using Google Clouds Speech API.

We will talk about each of these.

Using Android inbuilt Speech to Text function.

Android provides an inbuilt method to convert speech into text, it is the most easy way to convert speech to text.

This method uses android.speech package and a specific class called android.speech.RecognizerIntent . Basically we trigger an intent (android.speech.RecognizerIntent) which shows dialog box to recognize speech input. This Activity then converts the speech into text and send backs the result to our calling Activity. When we invoke android.speech.RecognizerIntent intent, we must use startActivityForResult()as we must listen back for result text.

The code snippet for the following is

private void promptSpeechInput() {
        Intent intent = new Intent(RecognizerIntent.ACTION_RECOGNIZE_SPEECH);
        intent.putExtra(RecognizerIntent.EXTRA_LANGUAGE_MODEL,
                RecognizerIntent.LANGUAGE_MODEL_FREE_FORM);
        intent.putExtra(RecognizerIntent.EXTRA_LANGUAGE, Locale.getDefault());
        intent.putExtra(RecognizerIntent.EXTRA_PROMPT,
                getString(R.string.speech_prompt));
        try {
            startActivityForResult(intent, REQ_CODE_SPEECH_INPUT);
        } catch (ActivityNotFoundException a) {
            Toast.makeText(getApplicationContext(),
                    getString(R.string.speech_not_supported),
                    Toast.LENGTH_SHORT).show();
        }
    }

In the above code as we can see that we putting some extra information while passing the intent. This information is used by speech to text engine to determine the language of the user. Thus while invoking RecognizerIntent, we must provide extra RecognizerIntent.EXTRA_LANGUAGE_MODE. Here we are setting its value to en-US.

Since the recognizer is triggered we receive a callback onActivityResult(int requestCode, int resultCode, Intent data) which is an override method to handle the result. The RecognizerIntent will convert the speech input to text and send back the result as ArraList with key RecognizerIntent.EXTRA_RESULTS. Generally this list should be ordered in descending order of speech recognizer confidence. Only present when RESULT_OK is returned in an activity result. We just set the text that we got in result in text view txtText using txtText.setText()

The screenshot of the implementation is

Using Google Cloud Speech API

Google Cloud Api is enable the developers to convert speech to text in Real time . It is used in Google Allo and Google Assistant. It is backed by powerful neural network and machine learning algorithms which makes it very efficient and fast at the same time. The Api is capable of recognizing more than 80 languages. To find more detail about Google Cloud Speech Api, one can refer to the official documentation at this link.

The Google to text Api is not free and based on the usage of the Api. To use this Api, developer have to sign up at the google console to generate the Api key. On enabling the speech API a json will be created.

protected void onActivityResult(int requestCode, int resultCode, Intent data) {
        super.onActivityResult(requestCode, resultCode, data);

        switch (requestCode) {
            case REQ_CODE_SPEECH_INPUT: {
                if (resultCode == RESULT_OK && null != data) {
                    ArrayList<String> result = data.getStringArrayListExtra(RecognizerIntent.EXTRA_RESULTS);
                    mVoiceInputTv.setText(result.get(0));
                }
                break;
            }

        }
    }

The whole implementation of the API can be found here.

Hotword Detection in Susi Android

Hotword detection is one of the coolest features in Android. Voice control has emerged as a popular method for interacting with smartphones and wearable devices. It allows the user to interact with the app without even touching the device in a much intuitive way. At the same time it’s difficult to implement them in the Android app and requires use of large number of resources. Let us dive deeper into this topic.

So, Firstly What is a Hotword?

Hotword is generally a phrase or a word that can be used to initiate a particular task in the app. The user needs to speak the phrase or the hotword which on detection makes a callback that can be used to carry out a particular task in the app.

Examples of hotword include “Ok Google” which is used to initiate Google assistant in the Android mobile phones.

Why is it difficult to implement hotword detection?

To enable hotword detection is a cumbersome task. There are different problems associated with it including:-

  1. Change in the accent of the speaker.
  2. Continuous task of recognizing the voice and then matching it with the hotword.
  3. The features needs to be run as a service in android which causes the drain of battery.
  4. Also it is a memory intensive task at the same time.

Let’s us look at the present techniques available for the hotword detection.

  1. The Android provides a class called as AlwaysOnHotwordDetector. This class can be used to detect the keywords inside the activity of an Android app. For more details you can visit this link.

The implementation goes like this

/**

* @param text The keyphrase text to get the detector for.

* @param locale The java locale for the detector.

* @param callback A non-null Callback for receiving the recognition events.

* @param voiceInteractionService The current voice interaction service.

* @param modelManagementService A service that allows management of sound models.

*

* @hide

*/


public AlwaysOnHotwordDetector(String text, Locale locale, Callback callback, KeyphraseEnrollmentInfo keyphraseEnrollmentInfo, IVoiceInteractionService voiceInteractionService, IVoiceInteractionManagerService modelManagementService) {

  mText = text;

  mLocale = locale;

  mKeyphraseEnrollmentInfo = keyphraseEnrollmentInfo;

  mKeyphraseMetadata = mKeyphraseEnrollmentInfo.getKeyphraseMetadata(text, locale);

  mExternalCallback = callback;

  mHandler = new MyHandler();

  mInternalCallback = new SoundTriggerListener(mHandler);

  mVoiceInteractionService = voiceInteractionService;

  mModelManagementService = modelManagementService;

  new RefreshAvailabiltyTask().execute();



}

2.  We can also detect hotword using Pocketsphinx library in Android. The library acts as a service in the Android app. It is quite efficient when it comes to battery consumption. The implementation goes like this:-    

recognizer = defaultSetup()
   .setAcousticModel(new File(assetsDir, "en-us-ptm"))
   .setDictionary(new File(assetsDir, 
"cmudict-en-us.dict"))
   .getRecognizer();
recognizer.addListener(this);



// Create keyword-activation search.
recognizer.addKeyphraseSearch(KWS_SEARCH, KEYPHRASE);

// Create grammar-based searches.
File menuGrammar = new File(assetsDir, "menu.gram");
recognizer.addGrammarSearch(MENU_SEARCH, menuGrammar);

// Next search for digits
File digitsGrammar = new File(assetsDir, "digits.gram");
recognizer.addGrammarSearch(DIGITS_SEARCH, digitsGrammar);

// Create language model search.
File languageModel = new File(assetsDir, "weather.dmp");
recognizer.addNgramSearch(FORECAST_SEARCH, languageModel);


recognizer.startListening(searchName);

Now Lets us look at how we are implementing it in Susi Android:

In Susi Android, we have used CMU’s Pocketsphinx library for the hotword detection. We are using “Hi Susi” as the hotword for the detection.

private SpeechRecognizer recognizer;

/* Keyword we are looking for to activate menu */

private static final String KEYPHRASE = "hi susi";

/* Named searches allow to quickly reconfigure the decoder */

private static final String KWS_SEARCH = "hi susi";



The function setupRecognizer is used to initialize the Recognizer and detect the keyphrase



private void setupRecognizer(File assetsDir) throws IOException {

// The recognizer can be configured to perform multiple searches

// of different kind and switch between them

recognizer = SpeechRecognizerSetup.defaultSetup()

.setAcousticModel(new File(assetsDir, "en-us-ptm"))

.setDictionary(new File(assetsDir, "cmudict-en-us.dict"))

.setRawLogDir(assetsDir)

.getRecognizer();

recognizer.addListener(this);

recognizer.addKeyphraseSearch(KWS_SEARCH, KEYPHRASE);

There is still some need of improvement towards this approach in Susi app as the success rate of detection of keyword is not very high. We can overcome this by training the model or having a set of strings similar to that of “Hi Susi” such as “Hii Susi” or “Hi Sushi”. This will increase the rate of detection.