Voice driven apps

Read more about the article Implementation of Speech UI in Susper

Implementation of Speech UI in Susper

Recently, we have implemented a speech recognition feature in Susper where user could search by voice but it does not have an attractive UI. Google has a good user experience while recording the voice. We have implemented a similar Speech UI in Susper,

How we have implemented this?

  1. First we made a component speechtotext. It takes care of all the styling and functional changes of the speech UI and rendering the speech and any instructions required for the user. https://github.com/fossasia/susper.com/tree/master/src/app/speechtotext
  2. Initially when user clicks on the microphone in the search bar, it triggers the speechRecognition()

searchbar.component.html

<div class="input-group-btn">
 <button class="btn btn-default" id="speech-button" type="submit">
   <img src="../../assets/images/microphone.png" class="microphone" (click)="speechRecognition()"/>
 </button>

searchbar.component.ts

speechRecognition() {
 this.store.dispatch(new speechactions.SearchAction(true));
}

3) This dispatches an action speechaction.SearchAction(true), the app.component.ts is subscribed to this action and whenever this action is triggered the app component will open the speechtotext component.

Speechtotext.component.ts

Speech to text component on getting initialised calls the speech service’s record function which activates standard browser’s speech API

constructor(private speech: SpeechService) {
 
 this.speechRecognition();
}
speechRecognition() {
 this.speech.record('en_US').subscribe(voice => this.onquery(voice));
}

On recording the user’s voice and converting it to text, the text is sent to the onquery method as input and the recognised text is sent to other components through ngrx store.

onquery(event: any) {
 this.resettimer();
 this.store.dispatch(new queryactions.QueryServerAction({ 'query': event, start: 0, rows: 10, search: true }));
 this.message = event;
}

We have some UI text transitions where the user is shown with messages like ‘Listening…’ ,‘Speak Now’ and ‘Please check your microphone’ which are handle by creating a timer observable in angular.

ngOnInit() {
 this.timer = Observable.timer(1500, 2000);
 this.subscription = this.timer.subscribe(t => {
   this.ticks = t;

   if (t === 1) {
     this.message = "Listening...";
   }
   if (t === 4) {
     this.message = "Please check your microphone and audio levels.";
     this.miccolor = '#C2C2C2';
   }
   if (t === 6) {
     this.subscription.unsubscribe();
     this.store.dispatch(new speechactions.SearchAction(false));
   }
 });
}

The related PR regarding speech to text is at https://github.com/fossasia/susper.com/issues/624 .

With this now we have achieved a good UI for handling requests on Speech.

Resources:

Continue ReadingImplementation of Speech UI in Susper