Implementing SUSI Linux App as a Finite State Machine

SUSI Linux app provides access to SUSI on Linux distributions on desktop as well as hardware devices like Raspberry Pi. It is a headless client that can be used to interact to SUSI via voice only. As more and more features like multiple hotword detection support and wake button support was added to SUSI Linux, the code became complex to understand and manage. A system was needed to model the app after. Finite State Machine is a perfect approach for such system.

The Wikipedia definition of State Machine is

It is an abstract machine that can be in exactly one of a finite number of states at any given time. The FSM can change from one state to another in response to some external inputs; the change from one state to another is called a transition.”

This means that if you can model your app into a finite number of states, you may consider using the State Machine implementation.

State Machine implementation has following advantages:

  • Better control over the working of the app.
  • Improved Error handling by making an Error State to handle errors.
  • States work independently which helps to modularize code in a better form.

To begin with, we declare an abstract State class. This class declares all the common properties of a state and transition method.

from abc import ABC, abstractclassmethod
import logging

class State(ABC):
   def __init__(self, components):
       self.components = components
       self.allowedStateTransitions = {}

   @abstractclassmethod
   def on_enter(self, payload=None):
       pass

   @abstractclassmethod
   def on_exit(self):
       pass

   def transition(self, state, payload=None):
       if not self.__can_transition(state):
           logging.warning("Invalid transition to State{0}".format(state))
           return

       self.on_exit()
       state.on_enter(payload)

   def __can_transition(self, state):
       return state in self.allowedStateTransitions.values()

We declared the on_enter() and on_exit() abstract method. These methods are executed on the entering and exiting a state respectively. The task designated for the state can be performed in the on_enter() method and it can free up resources or stop listening to callbacks in the on_exit() method. The transition method is to transition between one state to another. In a state machine, a state can transition to one of the allowed states only. Thus, we check if the transition is allowed or not before continuing it. The on_enter() and transition() methods additionally accepts a payload argument. This can be used to transfer some data to the state from the previous state.

We also added the components property to the State. Components store the shared components that can be used across all the State and are needed to be initialized only once. We create a component class declaring all the components that are needed to be used by states.

class Components:

   def __init__(self):
       recognizer = Recognizer()
       recognizer.dynamic_energy_threshold = False
       recognizer.energy_threshold = 1000
       self.recognizer = recognizer
       self.microphone = Microphone()
       self.susi = susi
       self.config = json_config.connect('config.json')

       if self.config['hotword_engine'] == 'Snowboy':
           from main.hotword_engine import SnowboyDetector
           self.hotword_detector = SnowboyDetector()
       else:
           from main.hotword_engine import PocketSphinxDetector
           self.hotword_detector = PocketSphinxDetector()

       if self.config['wake_button'] == 'enabled':
           if self.config['device'] == 'RaspberryPi':
               from ..hardware_components import RaspberryPiWakeButton
               self.wake_button = RaspberryPiWakeButton()
           else:
               self.wake_button = None
       else:
           self.wake_button = None

Now, we list out all the states that we need to implement in our app. This includes:

  • Idle State: App is listening for Hotword or Wake Button.
  • Recognizing State: App actively records audio from Microphone and performs Speech Recognition.
  • Busy State: SUSI API is called for the response of the query and the reply is spoken.
  • Error State: Upon any error in the above state, control transfers to Error State. This state needs to handle the speak the correct error message and then move the machine to Idle State.

Each state can be implemented by inheriting the base State class and implementing the on_enter() and on_exit() methods to implement the correct behavior.

We also declare a SUSI State Machine class to store the information about current state and declare the valid transitions for all the states.

class SusiStateMachine:
   def __init__(self):
       super().__init__()
       components = Components()
       self.__idle_state = IdleState(components)
       self.__recognizing_state = RecognizingState(components)
       self.__busy_state = BusyState(components)
       self.__error_state = ErrorState(components)
       self.current_state = self.__idle_state

       self.__idle_state.allowedStateTransitions = \
           {'recognizing': self.__recognizing_state, 'error': self.__error_state}
       self.__recognizing_state.allowedStateTransitions = \
           {'busy': self.__busy_state, 'error': self.__error_state}
       self.__busy_state.allowedStateTransitions = \
           {'idle': self.__idle_state, 'error': self.__error_state}
       self.__error_state.allowedStateTransitions = \
           {'idle': self.__idle_state}

       self.current_state.on_enter(payload=None)

We also set Idle State as the current State of the System. In this way, State Machine approach is implemented in SUSI Linux.

Resources:

 

Checking Image Size to Avoid Crash in Android Apps

In Giggity app a user can create a shortcut for the event by clicking on “home shortcut” button in the navigation drawer. Open Event format provides the logo URL in the return data so we do not need to provide it separately in the app’s raw file.

Sometimes the image can be too big to be put on screen as icon for shortcut. In this blog I describe a very simple method to check if we should use the image or not to avoid the crash and pixelation due to resolution.

We can store the image received in bitmap format. A bitmap is a type of memory organization or image file format used to store digital images. The term bitmap comes from the computer programming terminology, meaning just a map of bits, a spatially mapped array of bits. By storing it in bitmap format we can easily get the necessary information about the image to check if it is suitable for use.

We can use the BitmapFactory class which provides several decoding methods like (decodeByteArray(), decodeFile(), decodeResource(), etc.) for creating a Bitmap from various sources. Choose the most appropriate decode method based on your image data source. These methods attempt to allocate memory for the constructed bitmap and therefore can easily result in an OutOfMemory exception. Each type of decode method has additional signatures that let you specify decoding options via the BitmapFactory.Options class. Setting the inJustDecodeBounds property to true while decoding avoids memory allocation, returning null for the bitmap object but setting outWidth, outHeight and outMimeType. This technique allows you to read the dimensions and type of the image data prior to construction (and memory allocation) of the bitmap.

BitmapFactory.Options options = new BitmapFactory.Options();
options.inJustDecodeBounds = true;
BitmapFactory.decodeResource(getResources(), R.id.myimage, options);
int imageHeight = options.outHeight;
int imageWidth = options.outWidth;
String imageType = options.outMimeType;

To avoid java.lang.OutOfMemory exceptions, check the dimensions of a bitmap before decoding it, unless you absolutely trust the source to provide you with predictably sized image data that comfortably fits within the available memory.

So here is the particular example from Giggity app, it avoids crash on the recieving a large image for the icon. So once we store the the image in bitmap format we check if the height and width of the icon is exceeding the maximum limit.

public Bitmap getIconBitmap() {

 InputStream stream = getIconStream();
 Bitmap ret = null;

 if (stream != null) {
 ret = BitmapFactory.decodeStream(stream);
 if (ret == null) {
 Log.w("getIconBitmap", "Discarding unparseable file");
 return null;
 }
 if (ret.getHeight() > 512 || ret.getHeight() != ret.getWidth()) {
 Log.w("getIconBitmap", "Discarding, icon not square or >512 pixels");
 return null;
 }
 if (!ret.hasAlpha()) {
 Log.w("getIconBitmap", "Discarding, no alpha layer");
 return null;
 }
 }
 
 return ret;
}

If it does then we can avoid the icon. In this case we check if the icon is more than 512 pixels in height and width. If it is so then we could avoid it.

We could also check if the icon has a transparent background by using “hasAlpha” so we could have uniformity in the icons displayed on the screen. In the final result you can see the icon of the TUBIX 2017 conference added on the screen as it was following all those defined criterias.

Now that the image dimensions are known, they can be used to decide if the full image should be loaded into memory or if a subsampled version should be loaded instead. Here are some factors to consider:

  • Estimated memory usage of loading the full image in memory.
  • Amount of memory you are willing to commit to loading this image given any other memory requirements of your application.
  • Dimensions of the target ImageView or UI component that the image is to be loaded into.
  • Screen size and density of the current device.

For example, it’s not worth loading a 1024×768 pixel image into memory if it will eventually be displayed in a 128×96 pixel thumbnail in an ImageView.

 

References:

R14 – Memory Quota Exceeded

We, like many other organisations, are using heroku as the deployment server for our project open event organizer server. Things are pretty simple and awesome when your project is in its beginning phase and things run pretty smoothly. But as your project grows, there comes some server problem. And one of the biggest problems as your project grows is memory. Now since various packages have a different amount of memory assigned to you in case of hosting in generic servers such as heroku, so it might result in memory quota exceeded. Recently, we faced such a problem. R14 – Memory Quota Exceeded. Took us quite some time to understand what and why and how this occurred. So let me share a few things I found about this error.

Continue reading R14 – Memory Quota Exceeded

Error Handling in Retrofit 2

For the Open Event android app we were using retofit 1.9 with an okhttp stack plus a gson parser but recently retrofit 2.0 was released and it was a major update in the sense that it a lot of things have been changed.

For starters, you don’t have to declare synchronous and asynchronous requests upfront and you can just decide that while executing. The code for that will look something like this. This is how we define our request methods in our api service

import retrofit.Call;
public interface APIService {
   @POST(“/list”)
   Call<Repo> loadRepo();
}

Now if we want to make a synchronous request, we can make it like

Call<Repo> call = service.loadRepo();
Repo repo = call.execute();

and for an asynchronous request, we can call enqueue()

Call<Repo> call = service.loadRepo();
call.enqueue(new Callback<Repo>() {
    @Override
    public void onResponse(Response<Repo> response) {
    // Get result Repo from response.body()    
    }
    @Override
    public void onFailure(Throwable t) {

    }
});

And another thing that changed in the async call throws a throwable on failure, so essentially the RetrofitError class is gone and since we were using that in our app, we had to modify the whole error handling in the app, basically from the grounds up.

So, when we decided to move to retrofit 2 after the stable version was released, we had to change a lot of code and the main part that was affected was the error handling. So, replacing the retrofitError class, I used the throwable directly to retrieve the error type something like this

if (error.getThrowable() instanceof IOException) { 
    errorType = “Timeout”; 
    errorDesc = String.valueOf(error.getThrowable().getCause()); 
} 
else if (error.getThrowable() instanceof IllegalStateException) {                 
    errorType = “ConversionError”; 
    errorDesc = String.valueOf(error.getThrowable().getCause()); 
} else { 
    errorType = “Other Error”; 
    errorDesc = String.valueOf(error.getThrowable().getLocalizedMessage()); 
}

This was ofcourse for all failure events. And to handle all response events I compared the HTTP status codes and displayed the errors :

Integer statusCode = response.getStatusCode(); 
if (statusCode.equals(404)) { 
    // Show Errors in a dialog
    showErrorDialog(“HTTP Error”, statusCode + “Api Not Found”); 
}

This is how we can compare other HTTP errors in retrofit and assign the correct status accordingly. I personally think that this is a better implementation than Retrofit 1.9 and the RetrofitError was a bit tedious to work with. It wasn’t very thought of before implementation because it was not easy to tell what kind of error exactly occured. With Response codes, one can see what are the exact error one faces and can gracefully handle these errors.