Implementing Text To Speech Settings in SUSI WebChat

SUSI Web Chat has Text to Speech (TTS) Feature where it gives voice replies for user queries. The Text to Speech functionality was added using Speech Synthesis Feature of the Web Speech API. The Text to Speech Settings were added to customise the speech output by controlling features like :

  1. Language
  2. Rate
  3. Pitch

Let us visit SUSI Web Chat and try it out.

First, ensure that the settings have SpeechOutput or SpeechOutputAlways enabled. Then click on the Mic button and ask a query. SUSI responds to your query with a voice reply.

To control the Speech Output, visit Text To Speech Settings in the /settings route.

First, let us look at the language settings. The drop down list for Language is populated when the app is initialised. speechSynthesis.onvoiceschanged function is triggered when the app loads initially. There we call speechSynthesis.getVoices() to get the voice list of all the languages currently supported by that particular browser. We store this in MessageStore using ActionTypes.INIT_TTS_VOICES action type.

window.speechSynthesis.onvoiceschanged = function () {
  if (!MessageStore.getTTSInitStatus()) {
    var speechSynthesisVoices = speechSynthesis.getVoices();
    Actions.getTTSLangText(speechSynthesisVoices);
    Actions.initialiseTTSVoices(speechSynthesisVoices);
  }
};

We also get the translated text for every language present in the voice list for the text – `This is an example of speech synthesis` using google translate API. This is called initially for all the languages and is stored as translatedText attribute in the voice list for each element. This is later used when the user wants to listen to an example of speech output for a selected language, rate and pitch.

https://translate.googleapis.com/translate_a/single?client=gtx&sl=en-US&tl=TARGET_LANGUAGE_CODE&dt=t&q=TEXT_TO_BE_TRANSLATED

When the user visits the Text To Speech Settings, then the voice list stored in the MessageStore is retrieved and the drop down menu for Language is populated. The default language is fetched from UserPreferencesStore and the default language is accordingly highlighted in the dropdown. The list is parsed and populated as a drop down using populateVoiceList() function.

let voiceMenu = voices.map((voice,index) => {
  if(voice.translatedText === null){
    voice.translatedText = this.speechSynthesisExample;
  }
  langCodes.push(voice.lang);
  return(
    <MenuItem value={voice.lang}
              key={index}
              primaryText={voice.name+' ('+voice.lang+')'} />
  );
});

The language selected using this dropdown is only used as the language for the speech output when the server doesn’t specify the language in its response and the browser language is undefined. We then create sliders using Material UI for adjusting speech rate and pitch.

<h4 style={{'marginBottom':'0px'}}><Translate text="Speech Rate"/></h4>
<Slider
  min={0.5}
  max={2}
  value={this.state.rate}
  onChange={this.handleRate} />

The range for the sliders is :

  • Rate : 0.5 – 2
  • Pitch : 0 – 2

The default value for both rate and pitch is 1. We create a controlled slider saving the values in state and using onChange function to record change in values. The Reset buttons can be used to reset the rate and pitch values respectively to their default values. Once the language, rate and pitch values have been selected we can click on `Play a short demonstration of speech synthesis`  to listen to a voice reply with the chosen settings.

{ this.state.playExample &&
  (
    <VoicePlayer
       play={this.state.play}
       text={voiceOutput.voiceText}
       rate={this.state.rate}
       pitch={this.state.pitch}
       lang={this.state.ttsLanguage}
       onStart={this.onStart}
       onEnd={this.onEnd}
    />
  )
}

We use the VoicePlayer by passing the required props to get the speech output. onStart and onEnd functions are triggered at the beginning and ending of the speech synthesis and are used to control the state from the parent component. Chosen language, rate, pitch and translated text are passed as props to VoicePlayer which creates a new SpeechSynthesisUtterance() with the passed props and plays the speech output.

On saving these settings and then using the Mic button to get voice replies we see that the voice output is controlled according to the selected settings.

Finally, we have to store the selected settings on the server and ensure that these are pulled when the app is initialized. The format in which these settings are stored in the server is :

Speech Rate

- Used to control rate of speech output.
- SETTING_NAME :  `speechRate`
- SETTING_VALUE : `0.5 - 2`
- DEFAULT_VALUE : `1`
 
Speech Pitch

- Used to control pitch of speech output.
- SETTING_NAME :  `speechPitch`
- SETTING_VALUE : `0 - 2`
- DEFAULT_VALUE : `1`
 
TTS Language

- Used to set the language for Text-To-Speech used when the response from server doesnt specify language and the browser language is also undefined.
- SETTING_NAME :  `ttsLanguage`
- SETTING_VALUE : `Language Code (string)`
- DEFAULT_VALUE : `en-US`

This is how the Text To Speech Settings were implemented in SUSI Web Chat. The complete code can be found at SUSI Web Chat Repository.

PS: To test whether your browser supports Text To Speech, open your browser console and try the following :

  • var msg = new SpeechSynthesisUtterance(‘Hello World’);
  • window.speechSynthesis.speak(msg)

If you get a speech output then the Web API Speech Synthesis is supported by your browser and Text To Speech features of SUSI Web Chat will work. The Web Speech API has support for all latest Chrome browsers as mentioned in the Web Speech API Mozilla docs.However there are few bugs with some Chromium versions please check out more on how to fix them locally here in this link.

Resources:

 

 

Continue ReadingImplementing Text To Speech Settings in SUSI WebChat

Implementation of Text to Speech alongside Hotword Detection in SUSI Android App

In this blog post, we’ll be learning about how to implement Text to speech. Now you may be wondering that what is so difficult in implementing text to speech. One can easily find many tutorials on that and can easily look at the official documentation of TTS but there’s a catch here. In this blog post I’ll be telling about how to implement Text to Speech alongside Hotword Detection.

Let me give you a rough idea about how hotword detection works in SUSI Android App. For more details, read my other blog here on Hotword Detection. So, there is a constantly running background recording thread which detects when hotword is detected. Now, you may be thinking why do we need to stop that thread for text to speech. Well there are 2 reasons to do that:

  1. Recording while playing causing problems with mic and may crash the app.
  2. Suppose we even implement that but what will happen if the answer contains word “susi” in it. Now, the hotword will be detected because the speech output contained word “susi” in it (which is our hotword).

So, to avoid these problems we had to come up a way to stop hotword detection only for that particular time when SUSI is giving speech output and resume it back immediately when speech output is finished.

Let’s see how we did that.

Implementation

Check out this video to see how this work in the app

https://youtu.be/V9N6K4SzpXw

Initiating the TTS engine

The first task is to initiate the Text to speech engine. This process takes some time. So, it is done in the starting of app in a new handler.

new Handler().post(new Runnable() {
   @Override
   public void run() {
       textToSpeech = new TextToSpeech(getApplicationContext(), new TextToSpeech.OnInitListener() {
           @Override
           public void onInit(int status) {
               if (status != TextToSpeech.ERROR) {
                   Locale locale = textToSpeech.getLanguage();
                   textToSpeech.setLanguage(locale);
               }
           }
       });
   }
});

Check Audio Focus

The next step is to check whether audio focus is granted. Suppose there is some music playing in the background, in that case we won’t be able to give voice output. So, we check audio focus using below code.

final AudioManager audiofocus = (AudioManager) getSystemService(Context.AUDIO_SERVICE);
 int result = audiofocus.requestAudioFocus(afChangeListener, AudioManager.STREAM_MUSIC, AudioManager.AUDIOFOCUS_GAIN);
if (result == AudioManager.AUDIOFOCUS_REQUEST_GRANTED) {
//DO WORK HERE
}

Using OnAudioFocusChangeListener, we keep a track of when we have access to give speech output and when we don’t.

private AudioManager.OnAudioFocusChangeListener afChangeListener =
       new AudioManager.OnAudioFocusChangeListener() {
           public void onAudioFocusChange(int focusChange) {
               if (focusChange == AUDIOFOCUS_LOSS_TRANSIENT) {
                   textToSpeech.stop();
               } else if (focusChange == AudioManager.AUDIOFOCUS_GAIN) {
                   // Resume playback
               } else if (focusChange == AudioManager.AUDIOFOCUS_LOSS) {
                   textToSpeech.stop();
               }
           }
       };

Converting the given text to speech

Now we have audio focus, we just have to convert given text to speech. Use method textToSpeech.speak().

private void voiceReply(final String reply) {
       Handler handler = new Handler();
       handler.post(new Runnable() {
           @Override
           public void run() {
                   textToSpeech.speak(spokenReply, TextToSpeech.QUEUE_FLUSH, ttsParams);                  
               }
           }
       });
   }
}

Abandon Audio Focus

Now we are done with speech output, it’s time we abandon audio focus.

audiofocus.abandonAudioFocus(afChangeListener);

TTS alongside Hotword Detection

Okay so now the major part. How do we check when to stop hotword detection thread and when to resume it? How do we check if Speech output is finished?

Answer to these questions is textToSpeech.setOnUtteranceProgressListener. The UtteranceProgressListener overrides 3 methods:

  1. onStart: Indicates starting of text to speech conversion. Which means it’s time to stop hotword detection thread.
  2. onDone: Called when every word of the provided text is converted to speech. So, simply resume hotword detection
  3. onError: Called when there is an error and text is not converted to speech. Anyway, we need to resume hotword detection here too.
textToSpeech.setOnUtteranceProgressListener(new UtteranceProgressListener() {
                       @Override
                       public void onStart(String s) {
                           if(recordingThread !=null && isDetectionOn){
                               recordingThread.stopRecording();
                               isDetectionOn = false;
                           }
                       }

                       @Override
                       public void onDone(String s) {
                           if(recordingThread != null && !isDetectionOn && checkHotwordPref()) {
                               recordingThread.startRecording();
                               isDetectionOn = true;
                           }
                       }

                       @Override
                       public void onError(String s) {
                           if(recordingThread != null && !isDetectionOn && checkHotwordPref()) {
                               recordingThread.startRecording();
                               isDetectionOn = true;
                           }
                       }
                   });

                   HashMap<String,String> ttsParams = new HashMap<String, String>();
                   ttsParams.put(TextToSpeech.Engine.KEY_PARAM_UTTERANCE_ID,
                           MainActivity.this.getPackageName());

Summary

So, the main thing required for implementation of Text to Speech alongside Hotword detection is a way to control stopping and resuming hotword detection when Text to speech is in process. For that we used UtteranceProgressListener of TextToSpeech class which makes it so easier to do the task we required. You may follow this same approach as well or if you have a better approach, open an issue here.

Resources

  1. Official Documentation of TextToSpeech https://developer.android.com/reference/android/speech/tts/TextToSpeech.html
  2. Documentation of UtteranceProgressListener https://developer.android.com/reference/android/speech/tts/UtteranceProgressListener.html
  3. Blog link to Hotword Detection https://docs.google.com/document/d/1auTyuk32i15Rw94TOkrSruRJ9LZVtjcThoWVJkvnAz8/edit?usp=sharing
Continue ReadingImplementation of Text to Speech alongside Hotword Detection in SUSI Android App

Implementing Text to Speech on SUSI Web Chat

SUSI Web Chat now gives voice replies while chatting with it similar to SUSI Android and SUSI iOS clients. To test the Voice Output on Chrome,

  1. Visit chat.susi.ai
  2. Click on the Mic input button.
  3. Say something using the Mic when the Speak Now View appears

The simplest way to add text-to-speech to your website is by using the official Speech API currently available for Chrome Browser. The following steps help to achieve it in ReactJS.

  1. Initialize state defaults in a component named, VoicePlayer.
const defaults = {
      text: '',
      volume: 1,
      rate: 1,
      pitch: 1,
      lang: 'en-US'
    } 

There are two states which need to be maintained throughout the component and to be passed as such in our props which will maintain the state. Thus our state is simply

this.state = {
      started: false,
      playing: false
    }
  1. Our next step is to make use of functions of the Web Speech API to carry out the relevant processes.
  1. speak() – window.speechSynthesis.speak(this.speech) – Calls the speak method of the Speech API
  2. cancel() – window.speechSynthesis.cancel() – Calls the cancel method of the Speech API
  1. We then use our component helper functions to assign eventListeners and listen to any changes occurring in the background. For this purpose we make use of the functions componentWillReceiveProps(), shouldComponentUpdate(), componentDidMount(), componentWillUnmount(), and render().
  1. componentWillReceiveProps() – receives the object parameter {pause} to listen to any paused action
  2. shouldComponentUpdate() simply returns false if no updates are to be made in the speech outputs.
  3. componentDidMount() is the master function which listens to the action start, adds the eventListener start and end, and calls the speak() function if the prop play is true.
  4. componentWillUnmount() destroys the speech object and ends the speech.

Here’s a code snippet for Function componentDidMount()

componentDidMount () {
  
    const events = [
      { name: 'start', action: this.props.onStart }
    ]
    // Adding event listeners
    events.forEach(e => {
      this.speech.addEventListener(e.name, e.action)
    })

    this.speech.addEventListener('end', () => {
      this.setState({ started: false })
      this.props.onEnd()
    })

    if (this.props.play) {
      this.speak()
    }
  }
  1. We then add proper props validation in the following way to our VoicePlayer component.
VoicePlayer.propTypes = {
  play: PropTypes.bool,
  text: PropTypes.string,
  onStart: PropTypes.func,
  onEnd: PropTypes.func
};
  1. The next step is to pass the props from a listener view to the VoicePlayer component. Hence the listener here is the component MessageListItem.js from where the voice player is initialized.
  1. First step is to initialise the state.
this.state = {
      play: false,
    }
  onStart = () => {
    this.setState({ play: true });
  }
  onEnd = () => {
    this.setState({ play: false });
  }
  1. Next, we set play to true when we want to pass the props and the text which is to be said and append it to the message lists which have voice set as `true`
 { this.props.message.voice &&
      (<VoicePlayer
             play
             text={voiceOutput}
             onStart={this.onStart}
             onEnd={this.onEnd}
 />)}

Finally, our message lists with voice true will be heard on the speaker as they have been spoken on the microphone. To get access to the full code, go to the repository https://github.com/fossasia/chat.susi.ai or on our chat channel at gitter.im/fossasia/susi_webchat

Resources

  1. Speak-easy-synthesis repository http://mdn.github.io/web-speech-api/speak-easy-synthesis
  2. Web-speech-api repository https://github.com/mdn/web-speech-api/
Continue ReadingImplementing Text to Speech on SUSI Web Chat