Play Youtube Videos in SUSI.AI Android app

SUSI.AI Android app has many response functionalities ranging from giving simple ANSWER type responses to complex TABLE and MAP type responses. Although, even after all these useful response types there were some missing action types all related to media. SUSI.AI app was not capable of playing any kind of music or video.So, to do this  the largest source of videos in the world was thought and the functionality to play youtube videos in the app was added.

Since, the app now has two build flavors corresponding to the FDroid version and PlayStore version respectively it had to be considered that while adding the feature to play youtube videos any proprietary software was not included with the FDroid version. Google provides a youtube API that can be used to play videos inside the app only instead of passing an intent and playing the videos in the youtube app.

Steps to integrate youtube api in SUSI.AI

The first step was to create an environment variable that stores the API key, so that each developer that wants to test this functionality can just create an API key from the google API console and put it in the build.gradle file in the line below

def YOUTUBE_API_KEY = System.getenv(‘YOUTUBE_API_KEY’) ?: ‘”YOUR_API_KEY”‘    

In the build.gradle file the buildConfigfield parameter names API_KEY was created so that it can used whenever we need the API_KEY in the code. The buildConfigfield was declared for both the release and debug build types as :

release {
  minifyEnabled false
 proguardFiles getDefaultProguardFile(‘proguard-android.txt’), ‘proguard-rules.pro’
 buildConfigField “String”, ‘YOUTUBE_API_KEY’, YOUTUBE_API_KEY
}

debug {
  buildConfigField “String”, ‘YOUTUBE_API_KEY’, YOUTUBE_API_KEY
}

The second step is to catch the audio and video action type in the ParseSusiResponseHelper.kt file which was done by adding two constants “video_play” and “audio_play” in the Constant class. These actions were easily caught in the app as :

Constant.VIDEOPLAY -> try {
  identifier = susiResponse.answers[0].actions[i].identifier
} catch (e: Exception) {
  Timber.e(e)
}

Constant.AUDIOPLAY -> try {
  identifier = susiResponse.answers[0].actions[i].identifier
} catch (e: Exception) {
  Timber.e(e)
}

The third step involves making a ViewHolder class for the audio and video play actions. A simple layout was made for this viewholder which can display the thumbnail of the video and has a play button on the thumbnail, on clicking which plays the youtube video. Also, in the ChatFeedRecyclerAdapter we need to specify the action as Audio play and video play and then load the specific viewholder for the youtube videos whenever the response from the server for this particular action is fetched. The YoutubeViewHolder.java file describes the class that displays the thumbnail of the youtube video whenever a response arrives.

public void setPlayerView(ChatMessage model) {
  this.model = model;

  if (model != null) {
      try {
          videoId = model.getIdentifier();
          String img_url = “http://img.youtube.com/vi/” + videoId + “/0.jpg”;

          Picasso.with(itemView.getContext()).load(img_url).
                  placeholder(R.drawable.ic_play_circle_filled_white_24dp)
                  .into(playerView);
      } catch (Exception e) {
          Timber.e(e);
      }

  }

The above method shows how the thumbnail is set for a particular youtube video.

The fourth step is to pass the response from the server in the ChatPresenter to play the video asked by the user. This is  achieved by calling a function declared in the IChatView so that the youtube video will be played after fetching the response from the server :

if (psh.actionType == Constant.VIDEOPLAY || psh.actionType == Constant.AUDIOPLAY) {
  // Play youtube video
  identifier = psh.identifier
  chatView?.playVideo(identifier)
}

The fifth step is a bit complicated, as we know the app contains two flavors, one for the FDroid version that contains all the Open Source libraries and software and the second is the PlayStore version which may keep the proprietary software. Since, the youtube api is nothing but a piece of proprietary software we cannot include its implementation in the FDroid version and so different methods were devised to play the youtube videos in both versions. Since,  we use flavors and only want that Youtube API is compiled and included in the playstore version the youtube dependency was updated as :

playStoreImplementation files(‘libs/YouTubeAndroidPlayerApi.jar’)

This ensures that the library is only included in the playStore version. But on this step another problem is encountered, as now the code in both the versions will be different an encapsulation method was devised which enabled the app to keep separate code in both flavors.

To keep different code for both variants, in the src folder directories named fdroid and playStore were created and package name was added and then finally a java folder was created in each directory in which the separate code was kept for each flavor. An interface in the main directory was created which was used as a means to provide encapsulation so that it was implemented differently in each of the flavors to provide different functionality. The interface was created as :

interface IYoutubeVid {
  fun playYoutubeVid(videoId: String)
}

An object of the interface was made initialised in the ChatActivity and separate classes

named YoutubeVid.kt was made in each flavor which implemented the interface IYoutubeVid. So, now what happened was that depending on the build variant the YoutubeVid class of the particular flavor was called and the video would play according the sources that are available in that flavor.

In ChatActivity the following implementation was followed :

Declaration :

lateinit var youtubeVid: IYoutubeVid

Initialisation :

youtubeVid = YoutubeVid(this)

Call to the overridden function :

override fun playVideo(videoId: String) {
  Timber.d(videoId)
  youtubeVid.playYoutubeVid(videoId)
}

Final Output

References

  1. Youtube Android player API : https://developers.google.com/youtube/android/player/
  2. Building apps with product flavors – Tomoaki Imai https://medium.com/@tomoima525/building-multiple-regions-apps-with-product-flavors-good-and-bad-fe33e3c31060
  3. Kotlin style guide : https://android.github.io/kotlin-guides/style.html

 

Continue Reading

Added “table” type action support in SUSI android app

SUSI.AI has many actions supported by it, for eg: answer, anchor, map, piechart, websearch and rss.These actions are a few of those that can be supported in the SUSI.AI android app, but there are many actions implemented on the server side and the web client even has the implementation of how to handle the “table” type response.

The table response is generally a JSON array response with different json objects, where each json object have similar keys, and the actions key in the JSON response has the columns of the table response which are nothing but the keys in the data object of the response.

To implement the table type response in the susi android app a separate file needed to made to parse the table type response, since the keys and values both are required to the display the response. The file ParseTableSusiResponseHelper.kt was made which parsed the JSON object using the Gson converter factory to get the key value of the actions :

“actions”: [

       {

         “columns”: {

           “ingredients”: “Ingredients”,

           “href”: “Instructions Link”,

           “title”: “Recipe”

         },

         “count”: -1,

         “type”: “table”

       }

     ]

 

The inside the columns the keys and the values, both were extracted, values were to displayed in the title of the column and keys used were to extract the values from the “data” object of the response.

The files TableColumn.java, TableData.java are POJO classes that were used for storing the table columns and the data respectively. The TableDatas.java class was used to store the column list and the data list for the table response.

To fetch the table type response from the server a TableSusiResponse.kt file was added that contained serializable entities which were used to map the response values fetched from the server. A variable that contained the data stored in the “answers” key of the response was made of type of an ArrayList of TableAnswers.

@SerializedName(“answers”)
@Expose
val answers: List<TableAnswer> = ArrayList()

The TableAnswer.kt is another file added that contains serializable variables to store values inside the keys of the “answers” object. The actions object shown above is inside the answers object and it was stored in the form of an ArrayList of TableAction.

@SerializedName(“actions”)
@Expose
val actions: List<TableAction> = ArrayList()

Similar to TableAnswer.kt file TableAction.kt file also contains serializable variables that map the values stored in the “actions” object.

In the retrofit service interface SusiService.java a new call was added to fetch the data from the server as follows :

@GET(“/susi/chat.json”)
Call<TableSusiResponse> getTableSusiResponse(@Query(“timezoneOffset”) int timezoneOffset,
                                           @Query(“longitude”) double longitude,
                                           @Query(“latitude”) double latitude,
                                           @Query(“geosource”) String geosource,
                                           @Query(“language”) String language,
                                           @Query(“q”) String query);

Now, after the data was fetched, the table response can be parsed using the Gson converter factory in the ParseTableSusiResponseHelper.kt file. Below is the implementation :

fun parseSusiResponse(response: Response<TableSusiResponse>) {
  try {
      var response1 = Gson().toJson(response)
      var tableresponse = Gson().fromJson(response1, TableBody::class.java)
      for (tableanswer in tableresponse.body.answers) {
          for (answer in tableanswer.actions) {
              var map = answer.columns
              val set = map?.entries
              val iterator = set?.iterator()
              while (iterator?.hasNext().toString().toBoolean()) {
                  val entry = iterator?.next()
                  listColumn.add(entry?.key.toString())
                  listColVal.add(entry?.value.toString())
              }
          }
          val map2 = tableanswer.data
          val iterator2 = map2?.iterator()
          while (iterator2?.hasNext().toString().toBoolean()) {
              val entry2 = iterator2?.next()
              count++;
              for (count in 0..listColumn.size – 1) {
                  val obj = listColumn.get(count)
                  listTableData.add(entry2?.get(obj).toString())
              }
          }
          tableData = TableDatas(listColVal, listTableData)
      }
  } catch (e: Exception) {
      tableData = null
  }
}

 

Now the data is also parsed, we pass the two lists the ColumnList and DataList to the variable of TableDatas.

Three viewholder classes were added to display the table response properly in the app and corresponding to these viewholders a couple of adapters were also made that are responsible for setting the values in the recyclerview present in the views. The first viewholder is the TableViewHolder, it contains the horizontal recyclerview that is used to display the items fetched from the “data” object of the response. The recyclerview in the TableViewHolder has each entity of the type TabViewHolder, this is a simple cardview but also contains another recyclerview inside it which is used to store the keys and values of each of the object inside the “data” object.

TableViewHolder.java file has a setView() method that uses the the object of ChatMessage to get the list of columns and data to be set in the view.

 

Changes were made in the ChatPresenter.kt file to catch the tableresponse when a table type action is detected. Below is the implementation :

if (response.body().answers[0].actions[i].type.equals(“table”)) {
  tableResponse(query)
  return
}

The tableResponse function is as follows :

fun tableResponse(query: String) {
  val tz = TimeZone.getDefault()
  val now = Date()
  val timezoneOffset = -1 * (tz.getOffset(now.time) / 60000)
  val language = if (PrefManager.getString(Constant.LANGUAGE, Constant.DEFAULT).equals(Constant.DEFAULT)) Locale.getDefault().language else PrefManager.getString(Constant.LANGUAGE, Constant.DEFAULT)
  chatModel.getTableSusiMessage(timezoneOffset, longitude, latitude, source, language, query, this)
}

 

It calls the chatModel to get the list of columns and data to be set. The ChatFeedRecyclerAdapter.java files checks for the table response code, and if it matches then the view used for displaying SUSI’s message is the TableViewHolder. Here is how this viewholder is inflated :

case TABLE:
  view = inflater.inflate(R.layout.susi_table, viewGroup, false);
  return new TableViewHolder(view, clickListener);

Below is the final result when the table response is fetched for the query “Bayern munich team players” is :

References :

  1. SUSI server response for table query : https://api.susi.ai/susi/chat.json?timezoneOffset=-330&q=barcelona+team+players
  2. GSON for converting java objects to JSON and JSON to java : http://www.vogella.com/tutorials/JavaLibrary-Gson/article.html
Continue Reading

Link Preview Holder on SUSI.AI Android Chat

SUSI Android contains several view holders which binds a view based on its type, and one of them is LinkPreviewHolder. As the name suggests it is used for previewing links in the chat window. As soon as it receives an input as of link it inflates a link preview layout. The problem which exists was that whenever a user inputs a link as an input to app, it crashed. It crashed because it tries to inflate component that doesn’t exists in the view that is given to ViewHolder. So it gave a Null pointer Exception, due to which the app crashed. The work around for fixing this bug was that based on the type of user it will inflate the layout and its components. Let’s see how all functionalities were implemented in the LinkPreviewHolder class.

Components of LinkPreviewHolder

@BindView(R.id.text)
public TextView text;
@BindView(R.id.background_layout)
public LinearLayout backgroundLayout;
@BindView(R.id.link_preview_image)
public ImageView previewImageView;
@BindView(R.id.link_preview_title)
public TextView titleTextView;
@BindView(R.id.link_preview_description)
public TextView descriptionTextView;
@BindView(R.id.timestamp)
public TextView timestampTextView;
@BindView(R.id.preview_layout)
public LinearLayout previewLayout;
@Nullable @BindView(R.id.received_tick)
public ImageView receivedTick;
@Nullable
@BindView(R.id.thumbs_up)
protected ImageView thumbsUp;
@Nullable
@BindView(R.id.thumbs_down)
protected ImageView thumbsDown;

Currently in this it binds the view components with the associated id using declarator @BindView(id)

Instantiates the class with a constructor

public LinkPreviewViewHolder(View itemView , ClickListener listener) {
   super(itemView, listener);
   realm = Realm.getDefaultInstance();
   ButterKnife.bind(this,itemView);
}

Here it binds the current class with the view passed in the constructor using ButterKnife and initiates the ClickListener.

Now it is to set the components described above in the setView function:

Spanned answerText;
text.setLinksClickable(true);
text.setMovementMethod(LinkMovementMethod.getInstance());
if (android.os.Build.VERSION.SDK_INT >= Build.VERSION_CODES.N) {
answerText = Html.fromHtml(model.getContent(), Html.FROM_HTML_MODE_COMPACT);
} else {
answerText = Html.fromHtml(model.getContent());
}

Sets the textView inside the view with a clickable link. Version checking also has been put for checking the version of Android (Above Nougat or not) and implement the function accordingly.

This ViewHolder will inflate different components based on the thing that who has requested the output. If the query wants to inflate the LinkPreviewHolder them some extra set of components will get inflated which need not be inflated for the response apart from the basic layout.

if (viewType == USER_WITHLINK) {
   if (model.getIsDelivered())
       receivedTick.setImageResource(R.drawable.ic_check);
   else
       receivedTick.setImageResource(R.drawable.ic_clock);
}

In the above code  received tick image resource is set according to the attribute of message is delivered or not for the Query sent by the user. These components will only get initialised when the user has sent some links.

Now comes the configuration for the result obtained from the query.  Every skill has some rating associated to it. To mark the ratings there needs to be a counter set for rating the skills, positive or negative. This code should only execute for the response and not for the query part. This is the reason for crashing of the app because the logic tries to inflate the contents of the part of response but the view that is passed belongs to query. So it gives NullPointerException there, so there is a need to separate the logic of Response from the Query.

if (viewType != USER_WITHLINK) {
   if(model.getSkillLocation().isEmpty()){
       thumbsUp.setVisibility(View.GONE);
       thumbsDown.setVisibility(View.GONE);
   } else {
       thumbsUp.setVisibility(View.VISIBLE);
       thumbsDown.setVisibility(View.VISIBLE);
   }

   if(model.isPositiveRated()){
       thumbsUp.setImageResource(R.drawable.thumbs_up_solid);
   } else {
       thumbsUp.setImageResource(R.drawable.thumbs_up_outline);
   }

   if(model.isNegativeRated()){
       thumbsDown.setImageResource(R.drawable.thumbs_down_solid);
   } else {
       thumbsDown.setImageResource(R.drawable.thumbs_down_outline);
   }

   thumbsUp.setOnClickListener(new View.OnClickListener() {
       @Override
       public void onClick(View view) { . . . }
   });



   thumbsDown.setOnClickListener(new View.OnClickListener() {
       @Override
       public void onClick(View view) { . . . }
   });

}

As you can see in the above code  it inflates the rating components (thumbsUp and thumbsDown) for the view of the SUSI.AI response and set on the clickListeners for the rating buttons. Them in the below code it previews the link and commit the data using Realm in the database through WebLink class.

LinkPreviewCallback linkPreviewCallback = new LinkPreviewCallback() {
   @Override
   public void onPre() { . . . }

   @Override
   public void onPos(final SourceContent sourceContent, boolean b) { . . . }
}

This method calls the api and set the rating of that skill on the server. On successful result it made the thumb Icon change and alter the rating method and commit those changes in the databases using Realm.

private void rateSusiSkill(final String polarity, String locationUrl, final Context context) {..}

References

Continue Reading

Datewise splitting of the Bookmarks in the Homescreen of Open Event Android

In the Open Event Android app we had already incorporated bookmarks in the homescreen along with an improved UI. Now there was scope for further improvement in terms of user experience. The bookmarks were already sorted date wise but we needed to place them under separate date headers. In this blog I will be talking about how this was done in the app.

Initial Design
Current Design

 

 

 

 

 

 

 

 

Initially the user had no way of knowing which session belonged to which day. This could be fixed with a simple addition of a header indicating the day each bookmark belonged to. One way to do this was to add a day header and then get the bookmarks for each day and so on. But this proved to be difficult owing to the fact the number of days could be dynamic owing to the fact that this is a generic app. Another issue was that adding change listeners for the realm results to the bookmarks list for each day produced view duplication and other unexpected results whenever the bookmark list changed. So another approach was chosen that was to get all the bookmarks first and then add the date header and traverse through the bookmarks and only add sessions which belong to the date for which the date header was added earlier.

Bookmark Item Support in GlobalSearchAdapter

The main reason why we are reusing the GlobalSearchAdapter is that we have already defined a DIVIDER type in this adapter which can be reused as the date header.

We needed to initialize a constant for the Bookmark type.

private final int BOOKMARK = 5; //Can be any number

Then we add the Bookmark type in the getItemViewType() function which would return a constant that we defined earlier to indicate that in the filteredResultList we have an object of type Bookmark.

@Override
 public int getItemViewType(int position) {
    if (filteredResultList.get(position) instanceof Track) {
        return TRACK;
    }

    //Other Cases here
    } else if(filteredResultList.get(position) instanceof Session){
        return BOOKMARK;
    } else {
        return 1;
    }
 }

Now we create the viewholder if the list item is of the type Session which in this case will be a bookmark.

@Override
 public RecyclerView.ViewHolder onCreateViewHolder(ViewGroup parent, int viewType) {
    RecyclerView.ViewHolder resultHolder = null;
    LayoutInflater inflater = LayoutInflater.from(parent.getContext());
    //Other cases for Track,Location etc
 case BOOKMARK:
    View bookmark = inflater.inflate(R.layout.item_schedule, parent, false);
    resultHolder = new DayScheduleViewHolder(bookmark,context);
    break;
   //Some code
 
 }

Now we do the same in onBindViewHolder(). We bind the contents of the object to the ViewHolder here by calling the bindSession() function. We also pass in an argument which is our database repository i.e realmRepo here.

@Override
 public void onBindViewHolder(RecyclerView.ViewHolder holder, int position) {

    switch (holder.getItemViewType()) {
     //Other cases handled here

case BOOKMARK:
    DayScheduleViewHolder bookmarkTypeViewHolder = (DayScheduleViewHolder) holder;
    Session bookmarkItem = (Session) getItem(position);
    bookmarkTypeViewHolder.setSession(bookmarkItem);
    bookmarkTypeViewHolder.bindSession(realmRepo);
    break;
 }

Updating the AboutFragment

private GlobalSearchAdapter bookMarksListAdapter;
 private List<Object> mSessions = new ArrayList<>();

Earlier the DayScheduleAdapter was being used to display the list of bookmarks. Now we are reusing the GlobalSearchAdapter. Now we have also converted mSessions into a list of objects from a list of sessions.

Now we initialize the adapter so that we can start adding our date headers.

bookMarksListAdapter = new GlobalSearchAdapter(mSessions, getContext());
 bookmarksRecyclerView.setAdapter(bookMarksListAdapter);

In this function loadEventDates() we are storing the all the dates for the event. For example the list for the FOSSASIA17 sample stores the dates in the dateList as [2017-03-17,2017-03-18,2017-03-19]. We fetch the event dates by calling the getEventDateSync() function which has been defined in our Realm Database.

private void loadEventDates() {
 
    dateList.clear();
    RealmResults<EventDates> eventDates = realmRepo.getEventDatesSync();
    for (EventDates eventDate : eventDates) {
        dateList.add(eventDate.getDate());
    }
 }

Now we move on to the core logic of the feature which is to get the date headers to work correctly.

  • Fetch the list of bookmarks from the local Realm database asynchronously.
  • Remove any existing changeListeners to the bookmarkResult.
  • Add a changeListener to our list of results to notify us of the completion of the query or changes in the bookmark list.
  • After this is done, inside the changeListener we first clear the mSessions
  • We now traverse through our date list and compare it with the session startDate which we can obtain by calling the getStartDate(). If the date match occurs for the first time we add a date header after converting the date string into another format using the DateUtils class. So the function formatDay() of DateUtils converts 2017-03-17 to 17 Mar. This format is easily more readable.
  • Repeat for all dates.
private void loadData() {
    loadEventDates();
 
    bookmarksResult = realmRepo.getBookMarkedSessions();
    bookmarksResult.removeAllChangeListeners();
    bookmarksResult.addChangeListener((bookmarked, orderedCollectionInnerChangeSet) -> {
 
        mSessions.clear();
        for (String eventDate : dateList) {
            boolean headerCheck = false;
            for(Session bookmarkedSession : bookmarked){
                if(bookmarkedSession.getStartDate().equals(eventDate)){
                    if(!headerCheck){
                        String headerDate = "Invalid";
                      try {
                       headerDate = DateUtils.formatDay(eventDate);
                      }
                      catch (ParseException e){
                            e.printStackTrace();
                      }
                        mSessions.add(headerDate);
                        headerCheck = true;
                    }
                    mSessions.add(bookmarkedSession);
                }
            }
            bookMarksListAdapter.notifyDataSetChanged();
            handleVisibility();
        }
    });
 }

So, this is how the date-wise organization for the bookmarks in the homescreen was done.

Resources

Continue Reading
Close Menu