Handler

Implementation of Planned Actions in SUSI.AI Android Client

What do you understand by “Planned Actions”? Is it something required to do something according to plan? Is it something that needs to be done at a particular time? Yes, it is. “Planned Actions” is a feature present in the SUSI.AI android app that helps to schedule tasks. Yes, you heard it right! It helps to schedule the task. We can schedule some events like “setting an alarm after one minute”, “playing music after one minute”, etc. Such events require scheduling of different events at different point of time.

SUSI.AI is a smart assistant. It answers the queries that the user asks for, play music, etc. Additionally, the project also has a smart speaker part, which is very similar to Google Home. A smart speaker often has the ability to perform certain task other than answering normal queries at a scheduled time. During my GSoC period, I got the opportunity to implement such a feature of scheduling task in the SUSI.AI android app.

What is the scheduling of task or planned action?

Scheduling of task or planned actions is nothing but performing some kind of task at a scheduled time. Let’s consider “set alarm in one minute”. This means an alarm should be set that should ring up after one minute. 

Let’s see how it has been implemented in SUSI.

First of all, we need to have a skill that would respond to the query asked. Consider, the skill “rocket launch”. The response for this skill is:

{
  “query”: “rocket launch”,
  “client_id”: “aG9zdF8xNzIuNjguMTQ2LjE5MF80YTk0OGY4OA==”,
  “query_date”: “2019-08-19T17:31:53.670Z”,
  “answers”: [{
    “data”: [
      {
        “0”: “rocket launch”,
        “token_original”: “rocket”,
        “token_canonical”: “rocket”,
        “token_categorized”: “rocket”,
        “timezoneOffset”: “-330”,
        “language”: “en”
      },
      {
        “0”: “rocket launch”,
        “token_original”: “rocket”,
        “token_canonical”: “rocket”,
        “token_categorized”: “rocket”,
        “timezoneOffset”: “-330”,
        “language”: “en”
      }
    ],
    “metadata”: {“count”: 2},
    “actions”: [
      {
        “language”: “en”,
        “type”: “answer”,
        “expression”: “starting countdown”
      },
      {
        “expression”: “twelve”,
        “language”: “en”,
        “type”: “answer”,
        “plan_delay”: 1000,
        “plan_date”: “2019-08-19T17:31:54.679Z”
      },
      {
        “expression”: “eleven”,
        “language”: “en”,
        “type”: “answer”,
        “plan_delay”: 2000,
        “plan_date”: “2019-08-19T17:31:55.681Z”
      },
      {
        “expression”: “ten”,
        “language”: “en”,
        “type”: “answer”,
        “plan_delay”: 3000,
        “plan_date”: “2019-08-19T17:31:56.681Z”
      },
      {
        “expression”: “nine”,
        “language”: “en”,
        “type”: “answer”,
        “plan_delay”: 4000,
        “plan_date”: “2019-08-19T17:31:57.682Z”
      },
      {
        “expression”: “ignition sequence starts”,
        “language”: “en”,
        “type”: “answer”,
        “plan_delay”: 5000,
        “plan_date”: “2019-08-19T17:31:58.682Z”
      },
      {
        “expression”: “six”,
        “language”: “en”,
        “type”: “answer”,
        “plan_delay”: 7000,
        “plan_date”: “2019-08-19T17:32:00.682Z”
      },
      {
        “expression”: “five”,
        “language”: “en”,
        “type”: “answer”,
        “plan_delay”: 8000,
        “plan_date”: “2019-08-19T17:32:01.683Z”
      },
      {
        “expression”: “four”,
        “language”: “en”,
        “type”: “answer”,
        “plan_delay”: 9000,
        “plan_date”: “2019-08-19T17:32:02.683Z”
      },
      {
        “expression”: “three”,
        “language”: “en”,
        “type”: “answer”,
        “plan_delay”: 10000,
        “plan_date”: “2019-08-19T17:32:03.683Z”
      },
      {
        “expression”: “two”,
        “language”: “en”,
        “type”: “answer”,
        “plan_delay”: 11000,
        “plan_date”: “2019-08-19T17:32:04.684Z”
      },
      {
        “expression”: “one”,
        “language”: “en”,
        “type”: “answer”,
        “plan_delay”: 12000,
        “plan_date”: “2019-08-19T17:32:05.684Z”
      },
      {
        “expression”: “zero”,
        “language”: “en”,
        “type”: “answer”,
        “plan_delay”: 13000,
        “plan_date”: “2019-08-19T17:32:06.684Z”
      },
      {
        “expression”: “all engines running”,
        “language”: “en”,
        “type”: “answer”,
        “plan_delay”: 14000,
        “plan_date”: “2019-08-19T17:32:07.685Z”
      },
      {
        “expression”: “liftoff, we have a liftoff!”,
        “language”: “en”,
        “type”: “answer”,
        “plan_delay”: 16000,
        “plan_date”: “2019-08-19T17:32:09.685Z”
      }
    ],
    “skills”: [“/susi_skill_data/models/general/Novelty and Humour/en/Apollo_Countdown.txt”],
    “persona”: {}
  }],
  “answer_date”: “2019-08-19T17:31:53.685Z”,
  “answer_time”: 15,
  “language”: “en”,
  “session”: {“identity”: {
    “type”: “host”,
    “name”: “172.68.146.190_4a948f88”,
    “anonymous”: true
  }}
}

The response for the above skill can be received by querying “rocket launch” at https://api.susi.ai/  or you can get it by going directly to the following link https://api.susi.ai/susi/chat.json?timezoneOffset=-330&q=Rocket+Launch

The skill acts as a count-down timer of rocket launch. Now, we have a skill, but how are we going to implement it in android app, so that the countdown timer works properly and the output appears at the proper scheduled time. Also, we need to take care that the other processes of the app doesn’t stop or hangs up when some events are scheduled. Going through all these problems and circumstances I came up with a solution. So, let’s have a look. I made use of Handler. 

Handler in Android helps to execute some tasks at a scheduled time. The tasks are executed in a different thread other than the Main UI thread. So, the other process in the app doesn’t stops and it runs smoothly.

Let’s have a look at the code:

for (i in 0 until actionSize) {

  if (susiResponse.answers[0].actions[i].plan_delay.toString().isNullOrEmpty()) {
      planDelay = 0L
  } else {
      planDelay = susiResponse.answers[0].actions[i].plan_delay
  }
  executeTask(planDelay, susiResponse, i, date)
  chatView?.hideWaitingDots()
}

The code above, loops through all the actions mentioned in the skill one by one. After parsing it, the data is passed to a function executeTask(). This function performs the task to schedule the events (plan actions).

handler = Handler()
try {
  handler.postDelayed({
      val parseSusiHelper = ParseSusiResponseHelper()
      parseSusiHelper.parseSusiResponse(susiResponse, i, utilModel.getString(R.string.error_occurred_try_again))
      var setMessage = parseSusiHelper.answer

      if (parseSusiHelper.actionType == Constant.TABLE) {
          tableItem = parseSusiHelper.tableData
      } else if (parseSusiHelper.actionType == Constant.VIDEOPLAY || parseSusiHelper.actionType == Constant.AUDIOPLAY) {
          // Play youtube video
          identifier = parseSusiHelper.identifier
          youtubeVid.playYoutubeVid(identifier)
      } else if (parseSusiHelper.actionType == Constant.ANSWER && (PrefManager.checkSpeechOutputPref() && check || PrefManager.checkSpeechAlwaysPref())) {
          setMessage = parseSusiHelper.answer
          try {
              var speechReply = setMessage
              Handler().post {
                  textToSpeech = TextToSpeech(context, TextToSpeech.OnInitListener { status ->
                      if (status != TextToSpeech.ERROR) {
                          val locale = textToSpeech?.language
                          textToSpeech?.language = locale
                          textToSpeech?.speak(speechReply, TextToSpeech.QUEUE_FLUSH, null)
                          PrefManager.putBoolean(R.string.used_voice, true)
                      }
                  })
              }
          } catch (e: Exception) {
              Timber.e(“Error occured while trying to start text to speech engine – ” + e)
          }
      } else if (parseSusiHelper.actionType == Constant.STOP) {
          setMessage = parseSusiHelper.stop
          removeCallBacks()
          chatView?.stopMic()
      }

      if (parseSusiHelper.answer == ALARM) {
          playRingTone()
      }

      try {
          databaseRepository.updateDatabase(ChatArgs(
                  prevId = id,
                  message = setMessage,
                  date = DateTimeHelper.getDate(date),
                  timeStamp = DateTimeHelper.getTime(date),
                  actionType = parseSusiHelper.actionType,
                  mapData = parseSusiHelper.mapData,
                  isHavingLink = parseSusiHelper.isHavingLink,
                  datumList = parseSusiHelper.datumList,
                  webSearch = parseSusiHelper.webSearch,
                  tableItem = tableItem,
                  identifier = identifier,
                  skillLocation = susiResponse.answers[0].skills[0]
          ), this)
      } catch (e: Exception) {
          Timber.e(“Error occured while updating the database – ” + e)
          databaseRepository.updateDatabase(ChatArgs(
                  prevId = id,
                  message = utilModel.getString(R.string.error_internet_connectivity),
                  date = DateTimeHelper.date,
                  timeStamp = DateTimeHelper.currentTime,
                  actionType = Constant.ANSWER
          ), this)
      }
  }, planDelay)
} catch (e: java.lang.Exception) {
  Timber.e(“Error while showing data – ” + e)
}

The variable planDelay determines the time after which the task needs to be executed. By default its value is 0, i.e, it gets executed at that moment only until any delay time is supplied by the skill.

In this way, planned actions have been implemented in SUSI.AI Android app.

Resources: 

Blogs: Handler, Handler Implementation

SUSI Skill: Rocket Launch

SUSI.AI Android App: PlayStore GitHub

Tags:SUSI.AI Android App, Kotlin, SUSI.AI, FOSSASIA, GSoC, Android, Handler, Planned Actions

Continue ReadingImplementation of Planned Actions in SUSI.AI Android Client

Handler in Android

While working on SUSI.AI app (a smart assistant app), I found the necessity of handling UI components along with background data queue. While initializing a text to speech(TTS) engine inside a fragment that already had a speech to text (stt) engine implemented, there was a necessity to run the TTS engine to run using handler and later make it interact with the main UI thread. Let’s see how I handled this situation in the SUSI.AI app.

Android usually handles all the UI operations and input events from one single thread which is known as the Main or UI thread. Android collects all events in this thread in a queue and processes this queue. If one needs to perform some tasks in parallel with the main thread, then the main thread needs to be synchronized. Each handler is associated with a thread. A handler is a way to solve the asynchronous problem in Android.

A handler is widely used for:

  • Managing and inserting messages in the queue
  • Performing a task at a scheduled time in a different thread
  • Implementing runnable 

How Handlers work?

A handler is used to send and process Message and Runnable objects associated with a thread. Each handler is associated with a different single thread. This helps to perform the task asynchronously. The task, messages or runnable associated with that handlers are executed when it comes out of the message queue.

In Android, Handler is mainly used to update the main thread from background thread or other than the main thread. There are two methods in the handler:

  • Post() − it posts a message from background thread to the main thread using looper.
  • sendmessage() − if we want to organize what we have sent to UI (message from background thread) or UI functions. we should use sendMessage().

Construction of Handler:

First of all, we need to create and reference to Handler. After Handler is being created, we create some runnable objects. These runnable objects get executed inside the Handler.

Construction of Runnables that can be used in the handler:

Here, in this example, I have used the Post method to update the main thread.

Clearly, the post method is used in the handler object to execute the task mentioned in the runnable, by using its reference. Also, we can see the use of postDelayed function. This function executes the runnable after a specific span of time(mentioned in milliseconds in the parameter along with the runnable reference).

Lastly, it is very important to note that we should clear all the references to the handlers during the destruction of the view or the activity else there might be memory leaks. 

Resources: 

Documentation: Handler

Reference: Handler in Android

SUSI.AI Android App: PlayStore GitHub

Tags:SUSI.AI Android App, Kotlin, SUSI.AI, FOSSASIA, GSoC, Android, Handler

Continue ReadingHandler in Android