Adding Support for Playing Audio in SUSI iOS App

SUSI.AI supports various actions like the answer, map, table, video play and many more. You can play youtube videos in the chat screen. It also supports for playing audio in the chat screen. In this post, we will see that how playing audio feature implemented in SUSI iOS.

Getting audio action from server side –

In the chat screen, when we ask SUSI to play audio, we get the audio source from the server side. For example, if we ask SUSI “open the pod bay door”, we get the following action object:

"actions": [
{
"type": "audio_play",
"identifier_type": "youtube",
"identifier": "7qnd-hdmgfk"
},
{
"language": "en",
"type": "answer",
"expression": "I'm sorry, Dave. I'm afraid I can't do that."
}
]

In the above action object, we can see that we get two actions, audio_play and answer. In audio_play action, we are getting an identifier type which tells us about the source of audio. Identifier type can be youtube or local or any other source. When the identifier is youtube, we play audio from youtube stream. In identifier, we get the audio file path. In case of youtube identifier type, we get youtube video ID and play from youtube stream. In answer action type, we get the expression which we display in chat screen after thumbnail.

Implementing Audio Support in App –

We use Google’s youtube Iframe API to stream audio from youtube videos. We have a VideoPlayerView that handle all the iFrame API methods and player events with help of YTPlayer HTML file.

Presenting the YouTubePlayerCell –

If the action type is audio_play, we are presenting the cell in chat screen using cellForItemAt method of UICollectionView.

if message.actionType == ActionType.audio_play.rawValue {
if let cell = collectionView.dequeueReusableCell(withReuseIdentifier: ControllerConstants.youtubePlayerCell, for: indexPath) as? YouTubePlayerCell {
cell.message = message
cell.delegate = self
return cell
}
}

Setting size for cell –

Using sizeForItemAt method of UICollectionView to set the size.

if message.actionType == ActionType.audio_play.rawValue {
return CGSize(width: view.frame.width, height: 158)
}

In YouTubePlayerCell, we fetch thumbnail and display in the cell with a play button. On clicking the play button, we open the player and stream music.

Final Output –

Resources –

  1. Apple’s Documentations on sizeForItemAt
  2. SUSI API Sample for Audio Play Action
  3. YouTube iFrame API for iOS
  4. Apple’s Documentations on cellForItemAt
Continue Reading

Implementing news feature in loklak

The idea is to bring out an useful feature out of the enriched high quality of data provided by api.loklak.org. Through this blog post, I’ll be discussing about the implementation of a news feature similar to Google to provide latest news relevant to the query in loklak.

Creating news Component

First step in implementing news feature would be to create a news component which would be rendered on clicking of news tab.

ng g component feed/news

Creating an action & reducer for news status

This step involves creating an action & reducer function for tracking the status of news component on click of news tab. When the user will click on news tab, corresponding action would be dispatched to display the news component in feed.

import { Action } from ‘@ngrx/store’;
export const ActionTypes = {
    NEWS_STATUS: ‘[NEWS] NEWS STATUS’
};
export class NewsStatusAction implements Action {
    type = ActionTypes.NEWS_STATUS;
    constructor (public payload: boolean) { }
}
export type Actions
    = NewsStatusAction;

 

Corresponding reducer function for news status to store current news status would be:

export function reducer(state: State = initialState, 
   action: newsStatusAction.NewsStatusAction): State {
switch (action.type) {
    case newsStatusAction.ActionTypes.NEWS_STATUS: {
    const newsStatusPayload: boolean = action.payload;
    return Object.assign({}, state, {
                newsStatus: newsStatusPayload
            });
        }
        default: {
            return state;
        }
    }
}

 

I would not be getting into each line of code here, only the important portions would be discussed here.

Creating news-org to store the names of news organizations

A new file news-org would be created inside app/shared/ folder containing list of news organizations in an array newsOrgs from which we would extract and filter results.

export const newsOrgs = [
    ‘CNN’,
    ‘nytimes’
];

 

Note: This is a dynamic implementation of news service and hence, the list of news organizations can be easily modified/updated here.

We would also need to export the newsOrgs in the index file inside shared/ folder as:

export { newsOrgs } from ‘./news-org’;

Creating ngrx action, reducer & effect for news success

Now comes the main part. We would need an action to dispatch on successful response of news search and corresponding reducer to keep appending the async news results.

News action would be created following the simple ngrx composition as:

import { Action } from ‘@ngrx/store’;
import { ApiResponse } from ‘../models/api-response’;
export const ActionTypes = {
    NEWS_SEARCH_SUCCESS: ‘[NEWS] SEARCH SUCCESS’
};
export class NewsSearchSuccessAction implements Action {
    type = ActionTypes.NEWS_SEARCH_SUCCESS;
    constructor (public payload: ApiResponse) { }
}
export type Actions
    = NewsSearchSuccessAction;

 

One of the important parts of news implementation is the reducer function corresponding to the news success action. We would actually append the async results provided by news effect into the store.

import * as newsSuccessAction from ‘../actions/newsSuccess’;
import { ApiResponseResult } from ‘../models’;

export interface State {
newsResponse: ApiResponseResult[];
}
export const initialState: State = {
newsResponse: [],
};
export function reducer(state: State = initialState,
   action: newsSuccessAction.NewsSearchSuccessAction): State {
    switch (action.type) {
        case newsSuccessAction.ActionTypes.NEWS_SEARCH_SUCCESS: {
            // Appending the news result
            return Object.assign({}, state, {
                newsResponse: […action.payload.statuses]
            });
        }
        default: {
            return state;
        }
    }
}
export const getNewsResponse = (state: State) => state.newsResponse;

 

Now, we would create an effect which would actually use the search service to fetch results for the list of organizations one by one will keep dispatching a corresponding action to append the response into the store to be displayed in news feed. Instead of showing whole code for the effect, only the main logic is being displayed.

import { newsOrgs } from ‘../shared/news-org’;
...
@Injectable()
export class DisplayNewsEffects {
    @Effect({ dispatch: false })
        searchNews$: Observable<void> = this.actions$
        .pipe(
        ofType(
        newsAction.ActionTypes.NEWS_STATUS
        ),
        map((action) => {
            if (action[‘payload’]) {
                const orgs = newsOrgs;
                orgs.forEach((org) => {
                const searchServiceConfig:
                    SearchServiceConfig = 
                    new SearchServiceConfig();
                this.apiSearchService
                .fetchQuery(‘from:’
                  + org, searchServiceConfig)
                    .subscribe(response =>
                this.store$.dispatch(
                     new newsStatusAction
                     .NewsSearchSuccessAction(response)));
                });
            }
        })
    );

Filtering the news response inside news component

The basic idea is to filter the results from news response which contain the query and display them using news template on click of news tab on results page of loklak.

import { Query } from ‘../../models/query’;
import { Observable } from ‘rxjs’;
...
public query: string;
public query$: Observable<Query>;
...
ngOnInit() {
   const texts = [];
   this.store.select(fromRoot.getQuery).subscribe(res => 
       this.query = res.displayString);
   this.query$ = this.store.select(fromRoot.getQuery);
   this.store.select(fromRoot.getNewsResponse).subscribe(
       v => {
       for ( let i = 0; i < v.length; i++ ) {
           this.newsResponse.push(v[i]);
           if (v[i][‘text’].includes(this.query
               .replace(/\s/g, ”).toLowerCase())) {
               if (!texts.includes(v[i][‘text’]
                   .replace(/\s/g, ”).toLowerCase())) {
                   texts.push(v[i][‘text’]
                       .replace(/\s/g, ”).toLowerCase());
                   this.newsResponse.push(v[i]);
               }
           }
       }
   });
}

 

In above configuration, firstly we needed the current query which we extracted from ngrx store and then checking whether the text of news results contain the query string or not. If the query is present inside the text, we would add it into an array which would be used to display news feed using feed-card.

Displaying the news response

Displaying the news response is very easy, we just need to pass the news response items with index inside feed-card component which would display the news feed similar to the normal feed under the news section.

<div class=“wrapper feed-results”>
   <div *ngFor=“let item of newsResponse; let i = index “>
     <feed-card [feedItem]=“item” [feedIndex]=“i”></feed-card>
   </div>
</div>

Creating News tab to display news feed

To provide a tab to user to access news feed, we would need to create a News tab along with the other searching tools inside feed-advanced-search component as:

<button [class.selected]=“selectedTab === ‘news'” class=“tab” value=“news”
   (click)=“getFilterResults($event.currentTarget.value)”>
   News
</button>

 

Note: Complete code for news implementation can be found here and the news filter can be found here.

Testing news feature

Search a query on loklak, and then click on news tab to get the latest relevant news matching the query.

Resources

Continue Reading

“STOP” action in SUSI Android App

Generally whenever there was a long query asked from SUSI through speech, it would respond to the user with a speech output, similarly the output is given through speech whenever the user clicks on the “Try it” button on the skill details activity.

The long answers for e.g. asking SUSI “How to cook biryani ?” gives a very long response which when conveyed through the speech output takes a long amount of time. Also most of the time users don’t want to listen to such long answers, but at the same time there is no option or feature to stop SUSI. The user either needs to switch over to another activity or has to close the app.

So, to solve this problem such that neither the user has to shut down the app nor the user has to switch over to another activity, the “STOP” action was added in SUSI.

The “STOP” action was integrated in the server and is of following type :

 

“actions”: [{“type”: “stop”}],

How to define STOP response ?

To integrate this action type in the app, a separate response type was added and checked for. In the file ParseSusiResponseHandler.kt the action type stop was added as :

Constant.STOP -> try {
  stop = susiResponse.answers[0].actions[1].type
} catch (e: Exception) {

}

So, now whenever the query of type “stop” was entered by the user it would be caught by this block and further processing could be done. The stop variable takes the value from the JSON response fetched and the value extracted is the one stored against the type key in the second field of the actions JSON in the first field of the answers JSON.

What to be done when executing STOP?

After the STOP action was caught the task to do was to define the behaviour of the app when this response was caught. So, first thing that should happen when STOP is received is that the TextToSpeech Engine should close so that SUSI no longer can speak the sentence but defining top is more than just closing the TTS engine. STOP signifies that any activity that is being continued right now should be stopped.

So, Using the android lifecycle methods I added a function in the IChatView interface to define what actions should take place in the stopping process.

override fun stopMic() {
  onPause()
  registerReceiver(networkStateReceiver, IntentFilter(ConnectivityManager.CONNECTIVITY_ACTION))

  window.setSoftInputMode(
          WindowManager.LayoutParams.SOFT_INPUT_STATE_ALWAYS_HIDDEN
  )

  if (recordingThread != null)
      chatPresenter.startHotwordDetection()

  if (etMessage.text.toString().isNotEmpty()) {
      btnSpeak.setImageResource(R.drawable.ic_send_fab)
      etMessage.setText(“”)
      chatPresenter.micCheck(false)
  }

  chatPresenter.checkPreferences()
}

The above function sends the activity to the onPause() lifecycle method to put the activity in a pausing state so that all that is taking place right now in the activity stops and this definitely serves our purpose. But after pausing the activity, there was a further need to perform some functionality that had to done on the start of the activity and therefore the code for that was also added in this function.

How to catch STOP?

The below code was added in ChatPresenter.kt file in which if the actionType from the psh object of type ParseSusiResponseHelper is “STOP” then the view function stopMic() is called which was defined above.

val psh = ParseSusiResponseHelper()
psh.parseSusiResponse(susiResponse, i, utilModel.getString(R.string.error_occurred_try_again))

var setMessage = psh.answer
if (psh.actionType == Constant.ANSWER && (PrefManager.checkSpeechOutputPref() && check || PrefManager.checkSpeechAlwaysPref())) {
  setMessage = psh.answer

  var speechReply = setMessage
  if (psh.isHavingLink) {
      speechReply = setMessage.substring(0, setMessage.indexOf(“http”))
  }
  chatView?.voiceReply(speechReply, susiResponse.answers[0].actions[i].language)
} else if (psh.actionType == Constant.STOP) {
  setMessage = psh.stop
  chatView?.stopMic()
}

Final Output

References

  1. Stop  json response from susi server : https://api.susi.ai/susi/chat.json?timezoneOffset=-330&q=susi+stop
  2. Android life cycle methods – Google: https://developer.android.com/guide/components/activities/activity-lifecycle
  3. Interaction between view and presenters : https://medium.com/@cervonefrancesco/model-view-presenter-android-guidelines-94970b430ddf
Continue Reading

STOP action implementation in SUSI iOS

You may have experienced, you can stop Google home or Amazon Alexa during the ongoing task. The same feature is available for SUSI too. Now, SUSI can respond to ‘stop’ action and stop ongoing tasks (e.g. SUSI is narrating a story and if the user says STOP, it stops narrating the story). ‘stop’ action is introduced to enable the user to make SUSI stop anything it’s doing.

Video demonstration of how stop action work on SUSI iOS App can be found here.

Stop action is implemented on SUSI iOS, Web chat, and Android. Here we will see how it is implemented in SUSI iOS.

When you ask SUSI to stop, you get following actions object from server side:

"actions": [{"type": "stop"}]

Full JSON response can be found here.

When SUSI respond with ‘stop’ action, we create a new action type ‘stop’ and assign `Message` object `actionType` to ‘stop’.

Adding ‘stop’ to action type:

enum ActionType: String {
... // other action types
case stop
}

Assigning to the message object:

if type == ActionType.stop.rawValue {
message.actionType = ActionType.stop.rawValue
message.message = ControllerConstants.stopMessage
message.answerData = AnswerAction(action: action)
}

A new collectionView cell is created to respond user with “stoped” text.

Registering the stopCell:

collectionView?.register(StopCell.self, forCellWithReuseIdentifier: ControllerConstants.stopCell)

Add cell to the chat screen:

if message.actionType == ActionType.stop.rawValue {
if let cell = collectionView.dequeueReusableCell(withReuseIdentifier: ControllerConstants.stopCell, for: indexPath) as? StopCell {
cell.message = message
let message = ControllerConstants.stopMessage
let estimatedFrame = self.estimatedFrame(message: message)
cell.setupCell(estimatedFrame, view.frame)
return cell
}
}

AVFoundation’s AVSpeechSynthesizer API is used to stop the action:

func stopSpeakAction() {
speechSynthesizer.stopSpeaking(at: AVSpeechBoundary.immediate)
}

This method immediately stops the speak action.

Final Output:

Resources – 

  1. About SUSI: https://chat.susi.ai/overview
  2. JSON response for ‘stop’ action: https://api.susi.ai/susi/chat.json?timezoneOffset=-330&q=susi+stop
  3. AVSpeechSynthesisVoice: https://developer.apple.com/documentation/avfoundation/avspeechsynthesisvoice
  4. AVFoundation: https://developer.apple.com/av-foundation/
  5. SUSI iOS Link: https://github.com/fossasia/susi_iOS
  6. SUSI Android Link: https://github.com/fossasia/susi_android
  7. SUSI Web Chat Link: https://chat.susi.ai/
Continue Reading

Making Custom Change Listeners in PSLab Android

In this post we are going to learn how to make custom change listeners. There are many use cases for custom change listeners like if you want to initiate some action when some variable’s value is changed. In PSLab android app, this was required during initialisation of PSLab hardware device, it takes about 3-4 seconds to initialise the device which includes reading calibration data from device and process it. So before starting the initialisation process, app notifies user with the message, “Initialising Wait …” and after initialisation is done, user is notified with the message “Initialisation Completed”.

There might be other ways to accomplish this but I found making a custom change listener for boolean and trigger notifying user action on change of boolean value to be most organised way to do it.

Another way I can think of is to pass the fragment reference to the class  constructor for which the object is to be made. And Views need to be made public for access from that object to change status after some work is done.

Let’s look at an example, we would change status in a fragment after some task in object instantiation is completed.

Implementation

Class with variable on which custom change listener is required:
Create a class and declare a variable for which you want to listen the value change to trigger some action. In this example we have created a InitializationVariable class and defined a boolean variable named initialised.

Define an interface inside the class and that’s where the trick lies. When you set/change the value of the variable through a function setVariable(boolean value) in this case, note that we are triggering the interface method too.

public class InitializationVariable {

   public boolean initialised = false;
   private onValueChangeListener valueChangeListener;

   public boolean isInitialised() {
       return initialised;
   }

   public void setVariable(boolean value) {
       initialised = value;
       if (valueChangeListener != null) valueChangeListener.onChange();
   }

   public onValueChangeListener getValueChangeListener() {
       return valueChangeListener;
   }

   public void setValueChangeListener(onValueChangeListener valueChangeListener) {
       this.valueChangeListener = valueChangeListener;
   }

   public interface onValueChangeListener {
       void onChange();
   }

}

Create an object of above class in activity/fragment:
Create an object to the class we just made and attach onValueChangeListener to it. This example shows how it’s used in PSLab Android, you can use it anywhere but remember to access view elements from a valid context.

public static InitializationVariable booleanVariable;
public class HomeFragment extends Fragment {

   @BindView(R.id.tv_initialisation_status)
   TextView tvInitializationStatus;

   public static InitializationVariable booleanVariable;// object whose value change is noted

   public static HomeFragment newInstance() {
       HomeFragment homeFragment = new HomeFragment();
       return homeFragment;
   }

   @Nullable
   @Override
   public View onCreateView(LayoutInflater inflater, @Nullable ViewGroup container, @Nullable Bundle savedInstanceState) {
       View view = inflater.inflate(R.layout.home_fragment, container, false);
       unbinder = ButterKnife.bind(this, view);
       return view;
   }

   @Override
   public void onViewCreated(View view, @Nullable Bundle savedInstanceState) {
       super.onViewCreated(view, savedInstanceState);

       booleanVariable.setValueChangeListener(new InitializationVariable.onValueChangeListener() {
           @Override
           public void onChange() {
               if (booleanVariable.isInitialised())
                   tvInitializationStatus.setText("Initialsation Completed");
               else
                   tvInitializationStatus.setText("Initialising Wait ...");
           }
       });
  }
}

Now whenever booleanVariable.setVariable(value) is called, it triggers the onValueChangeListener where you can manage the action you wanted to do on value change.
This is similar to how other listeners are implemented .You implement an interface and call those interface methods on some value change and classes which implement those interface have overridden methods which handle the action after change.

Hopefully this post gives you an insight about how change listeners are implemented.

Note: This post was specific to PSLab Android App, you can create custom change listener on any variable in any class and perform action on value of the variable getting changed.

Resources

Continue Reading
Close Menu