Implementation of React Routers in SUSI Web Chat

When we were developing the SUSI Web Chat application we wanted to implement set of static pages with the chat application. In the start we just wanted to navigate  through different static pages and move back to the web chat application. But it takes time to load a new page when user clicks on a link. Our goal was therefore to minimize the loading time by using lazy loading. For that we used react-route .It is standard library for react js.

From the docs:

“React Router keeps your UI synced with the URL. It has a simple API with powerful features like lazy code loading, dynamic route matching, and location transition handling built right in. Make the URL your first thought, not an after-thought.” (https://www.npmjs.com/package/react-router-plus)

We need react-route to be installed in our application first. We can install it using NPM by running this command on project folder.

npm install --save react-router-dom

Next we have to set up our routes. We have two types of headers in our application. One is chat application header, second one is static page header. In static page header we have a navigation to switch between static pages.
First we need to choose the router type because there are two types of routers in react. “” and “” we can use “” in our example because our server can handle dynamic requests. If we are requesting data from static page we should use “” .
We used that in “” and made another new component called “” and used it on “index.js” like this.

import { BrowserRouter as Router } from 'react-router-dom';
Import App from .App;
 ReactDOM.render(
  	<IntlProvider locale={defaultPrefLanguage}>
		<Router> <App /> </Router>
	</IntlProvider>,
  	document.getElementById('root')  );

In “App.js” we can set up routes like this.

        <Switch>
            <Route exact path='/' component={ChatApp}/>
            <Route exact path="/overview" component={Overview} />
            <Route exact path='/blog' component={Blog} />
            <Route exact path="/logout" component={Logout} />
            <Route exact path="/settings" component={Settings} />
            <Route exact path="*" component={NotFound} />
        </Switch>

We use elements to render component when they match with the corresponding path. We use “path” to add router location that we need to match width the component. We use “exact” property to render the component if location exactly matches with the “path”. If we do not use “exact” property it renders when we have child routes after the path like “/blog/1 “ .
We used “” element to group routes.
We can’t use anchor () tags to navigate pages without reloading. We have to use tags instead of that. We have to replace all the places we have used

<a href= ‘URL’>lable name </a>

with this,

<Link to=’URL’>Lable name</Link>   

After doing above changes application will perform faster and it will load all page contents soon after you click the navigation links.

If you would like to join with FOSSASIA and contribute to SUSI Web Chat Application please fork this repository on Github.

Resources

Implementing the Feedback Functionality in SUSI Web Chat

SUSI AI now has a feedback feature where it collects user’s feedback for every response to learn and improve itself. The first step towards guided learning is building a dataset through a feedback mechanism which can be used to learn from and improvise the skill selection mechanism responsible for answering the user queries.

The flow behind the feedback mechanism is :

  1. For every SUSI response show thumbs up and thumbs down buttons.
  2. For the older messages, the feedback thumbs are disabled and only display the feedback already given. The user cannot change the feedback already given.
  3. For the latest SUSI response the user can change his feedback by clicking on thumbs up if he likes the response, else on thumbs down, until he gives a new query.
  4. When the new query is given by the user, the feedback recorded for the previous response is sent to the server.

Let’s visit SUSI Web Chat and try this out.

We can find the feedback thumbs for the response messages. The user cannot change the feedback he has already given for previous messages. For the latest message the user can toggle feedback until he sends the next query.

How is this implemented?

We first design the UI for feedback thumbs using Material UI SVG Icons. We need a separate component for the feedback UI because we need to store the state of feedback as positive or negative because we are allowing the user to change his feedback for the latest response until a new query is sent. And whenever the user clicks on a thumb, we update the state of the component as positive or negative accordingly.

import ThumbUp from 'material-ui/svg-icons/action/thumb-up';
import ThumbDown from 'material-ui/svg-icons/action/thumb-down';

feedbackButtons = (
  <span className='feedback' style={feedbackStyle}>
    <ThumbUp
      onClick={this.rateSkill.bind(this,'positive')}
      style={feedbackIndicator}
      color={positiveFeedbackColor}/>
    <ThumbDown
      onClick={this.rateSkill.bind(this,'negative')}
      style={feedbackIndicator}
      color={negativeFeedbackColor}/>
  </span>
);

The next step is to store the response in Message Store using saveFeedback Action. This will help us later to send the feedback to the server by querying it from the Message Store. The Action calls the Dispatcher with FEEDBACK_RECEIVED ActionType which is collected in the MessageStore and the feedback is updated in the Message Store.

let feedback = this.state.skill;

if(!(Object.keys(feedback).length === 0 &&    
feedback.constructor === Object)){
  feedback.rating = rating;
  this.props.message.feedback.rating = rating;
  Actions.saveFeedback(feedback);
}

case ActionTypes.FEEDBACK_RECEIVED: {
  _feedback = action.feedback;
  MessageStore.emitChange();
  break;
}

The final step is to send the feedback to the server. The server endpoint to store feedback for a skill requires other parameters apart from feedback to identify the skill. The server response contains an attribute `skills` which gives the path of the skill used to answer that query. From that path we need to parse :

  • Model : Highest level of abstraction for categorising skills
  • Group : Different groups under a model
  • Language : Language of the skill
  • Skill : Name of the skill

For example, for the query `what is the capital of germany` , the skills object is

"skills": ["/susi_skill_data/models/general/smalltalk/en/English-Standalone-aiml2susi.txt"]

So, for this skill,

    • Model : general
    • Group : smalltalk
    • Language : en
    • Skill : English-Standalone-aiml2susi

The server endpoint to store feedback for a particular skill is :

BASE_URL+'/cms/rateSkill.json?model=MODEL&group=GROUP&language=LANGUAGE&skill=SKILL&rating=RATING'

Where Model, Group, Language and Skill are parsed from the skill attribute of server response as discussed above and the Rating is either positive or negative and is collected from the user when he clicks on feedback thumbs.

When a new query is sent, the sendFeedback Action is triggered with the required attributes to make the server call to store feedback on server. The client then makes an Ajax call to the rateSkill endpoint to send the feedback to the server.

let url = BASE_URL+'/cms/rateSkill.json?'+
          'model='+feedback.model+
          '&group='+feedback.group+
          '&language='+feedback.language+
          '&skill='+feedback.skill+
          '&rating='+feedback.rating;

$.ajax({
  url: url,
  dataType: 'jsonp',
  crossDomain: true,
  timeout: 3000,
  async: false,
  success: function (response) {
    console.log(response);
  },
  error: function(errorThrown){
    console.log(errorThrown);
  }
});

This is how the feedback feedback mechanism works in SUSI Web Chat. The entire code can be found at SUSI Web Chat Repository.

Resources

 

Storing SUSI Settings for Anonymous Users

SUSI Web Chat is equipped with a number of settings by which it can be configured. For a user who is logged in and using the chat, he can change his settings and save them by sending it to the server. But for an anonymous user, this feature is not available while opening the chat every time, unless one stores the settings. Thus to overcome this problem we store the settings in the browser’s cookies which we do by following the steps.

The current User Settings available are :-

  1. Theme – One can change the theme to either ‘dark’ or ‘light’.
  2. To have the option to send a message on enter.
  3. Enable mic to give voice input.
  4. Enable speech output only for speech input.
  5. To have speech output always ON.
  6. To select the default language.
  7. To adjust the speech output rate.
  8. To adjust the speech output pitch.

The first step is to have a default state for all the settings which act as default settings for all users when opening the chat for the first time. We get these settings from the User preferences store available at the file UserPreferencesStore.js

let _defaults = {
                Theme: 'light',
                Server: 'http://api.susi.ai',
                StandardServer: 'http://api.susi.ai',
                EnterAsSend: true,
                MicInput: true,
                SpeechOutput: true,
               SpeechOutputAlways: false,
               SpeechRate: 1,
               SpeechPitch: 1,
           };

We assign these default settings as the default constructor state of our component so that we populate them beforehand.

2. On changing the values of the various settings we update the state through various function handlers which are as follows-

handleSelectChange – handles all the theme changes, handleEnterAsSendhandleMicInput – handles Mic Input as true, handleSpeechOutput – handles whether Speech Output should be enabled or not, handleSpeechOutputAlways – handles if user wants to listen to speech after every input, handleLanguage – handles the language settings related to speech, handleTextToSpeech – handles the Text to Speech Settings including Speech Rate, Pitch.        

  1. Once the values are changed we make use of the universal-cookie package and save the settings in the User’s browser. This is done in a function handleSubmit in the Settings.react.js file.

        

 import Cookies from 'universal-cookie';
     const cookies = new Cookies(); // Create cookie object.
     // set all values
     let vals = {
            theme: newTheme,
            server: newDefaultServer,
            enterAsSend: newEnterAsSend,
            micInput: newMicInput,
            speechOutput: newSpeechOutput,
            speechOutputAlways: newSpeechOutputAlways,
            rate: newSpeechRate,
            pitch: newSpeechPitch,
      }

    let settings = Object.assign({}, vals);
    settings.LocalStorage = true;
    // Store in cookies for anonymous user
    cookies.set('settings',settings);   
  1. Once the values are set in the browser we load the new settings every time the user opens the SUSI Web Chat by having the following checks in the file API.actions.js. We get the values from the cookies using the get function if the person is not logged in and initialise the new settings for that user.
if(cookies.get('loggedIn')===null||
    cookies.get('loggedIn')===undefined){
    // check if not logged in and get the value from the set cookie
    let settings = cookies.get('settings');
    if(settings!==undefined){
      // Check if the settings are set in the cookie
      SettingsActions.initialiseSettings(settings);
    }
  }
else{
// call the AJAX for getting Settings for loggedIn users
}

Resources

Handling Offline Message Responses in SUSI Web Chat

Previously, the SUSI Web Chat stopped working when there was no Internet connectivity. The application’s overall state was disturbed as one would see the loading message gif and the users were left to wonder on how to proceed. To handle this situation, we required notifying the User in the offline mode, with a message, that there is no Internet Connectivity.

This following image demonstrates the previous state where the application hung.

This image shows how this state was handled currently. One can test this out on http://chat.susi.ai by disconnecting and sending a message to SUSI and then connecting back to the Internet.

To achieve this, one needed to handle the offline and online events of the browser. The following steps can be followed to achieve this.

  1. We make use of the following eventListener functions to know whether the user is connected to the Internet.
// handles the Offlines event
window.addEventListener('offline', handleOffline.bind(this));  

// handles the Online event
window.addEventListener('online', handleOnline.bind(this));
  1. We then set a global offline message which is modified on the connections switching from online to an offline state. They are handled by the following functions.
let offlineMessage = null;

function handleOffline() {
  offlineMessage = 'Sorry, cannot answer that now. I have no net connectivity';
}
function handleOnline() {
  offlineMessage = null;
}
  1. We then handle the action createSUSIMessage() in API.actions.js and send the  AJAX request which we are making according to the offline/online state. This enables us to send the correct message response to the User even in the offline state and not letting the Application state crash.
// So if the offlineMessage variable is not null we call the AJAX 
if(!offlineMessage){
        // handle AJAX
}
else {
    // we create a message saying there is no Internet connectivity.
}
     
  1. The messages on refreshing back restore back to the original state as these are not being stored in the server. Hence the User is able to see the correct History, i.e., only those messages which were sent to the server and successfully responded to by SUSI.

Resources

Adding a Scroll To Bottom button in SUSI WebChat

SUSI Web Chat now has a scroll-to-bottom button which helps the users to scroll the app automatically to the bottom of the scroll area on button click. When the chat history is lengthy and the user has to scroll down manually it results in a bad UX. So the basic requirements of this scroll-to-bottom button are:

  1. The button must only be displayed when the user has scrolled up the message section
  2. On clicking the scroll-to-bottom button, the scroll area must be automatically scrolled to bottom.

Let’s visit SUSI Web Chat and try this out.

The button is not visible until there are enough messages to enable scrolling and the user has scrolled up. On clicking the button, the app automatically scrolls to the bottom pointing to the most recent message.

How was this implemented?

We first design our scroll-to-bottom button using Material UI  Floating Action Button and SVG Icons.

import FloatingActionButton from 'material-ui/FloatingActionButton';
import NavigateDown from 'material-ui/svg-icons/navigation/expand-more';

The button needs to be styled to be displayed at a fixed position on the bottom right corner of the message section. Positioning it on top of MessageSection above the MessageComposer, the button is also aligned with respect to the edges.

const scrollBottomStyle = {
  button : {
    float: 'right',
    marginRight: '5px',
    marginBottom: '10px',
    boxShadow:'none',
  },
  backgroundColor: '#fcfcfc',
  icon : {
    fill: UserPreferencesStore.getTheme()==='light' ? '#90a4ae' : '#7eaaaf'
  }
}

The button must only be displayed when the user has scrolled up. To implement this we need a state variable showScrollBottom which must be set to true or false accordingly based on the scroll offset.

{this.state.showScrollBottom &&
  <div className='scrollBottom'>
    <FloatingActionButton mini={true}
      style={scrollBottomStyle.button}
      backgroundColor={scrollBottomStyle.backgroundColor}
      iconStyle={scrollBottomStyle.icon}
      onTouchTap={this.forcedScrollToBottom}>
      <NavigateDown />
    </FloatingActionButton>
  </div>
}

Now we have to set our state variable showScrollBottom corresponding to the scroll offset. It must be set to true is the user has scrolled up and false if the scrollbar is already at the bottom. To implement this we need to listen to the scrolling events. We used react-custom-scrollbars for the scroll area wrapping the message section. We can listen to the scrolling events using the onScroll props. We also need to tag the scroll area using refs to access the scroll area instead of using findDOMNode as it is being deprecated.

import { Scrollbars } from 'react-custom-scrollbars';

<Scrollbars
  ref={(ref) => { this.scrollarea = ref; }}
  onScroll={this.onScroll}
>
  {messageListItems}
</Scrollbars>

Now, whenever a scroll action is performed, the onScroll() function is triggered. We now have to know if the scroll bar is at the bottom or not. We make use of the scroll area’s props to get the scroll offsets. The getValues() function returns an object containing different scroll offsets and scroll area dimensions. We are interested in values.top which tells about the scroll-top’s progress from 0 to 1 i.e when the scroll bar is at the top most point values.top is 0 and when its on the bottom most point, values.top is 1. So whenever values.top is 1, showScrollBottom is false else true.

onScroll = () => {
  let scrollarea = this.scrollarea;
  if(scrollarea){
    let scrollValues = scrollarea.getValues();
    if(scrollValues.top === 1){
      this.setState({
        showScrollBottom: false,
      });
    }
    else if(!this.state.showScrollBottom){
      this.setState({
        showScrollBottom: true,
      });
    }
  }
}

Finally, we need to scroll the chat app to the bottom on button click. Whenever showScrollBottom is updated, the state is changed, so componentDidUpdate is triggered which calls the _scrollToBottom() function. But we should change this to avoid scrolling to bottom on showScrollBottom update and the user is intending to scroll here. We use the function forcedScrollToBottom to be triggered on clicking the scroll-to-bottom button, which resets the scrollTop value to the height of the scroll area, thus pointing the scrollbar to the bottom.

forcedScrollToBottom = () => {
  let ul = this.scrollarea;
  if (ul) {
    ul.scrollTop(ul.getScrollHeight());
  }
}

We don’t have to worry about resetting showScrollBottom on forced scroll to bottom as the scrolling will trigger the onScroll function where the showScrollBottom state is handled accordingly.

This is how the scroll to bottom button has been implemented in SUSI Web Chat. The entire code can be found at SUSI Web Chat Repository.

Resources

 

Implementation of SUSI Web Chat Auto Sizing Message Composer

While we are using SUSI Web Chat Application we may have to send lengthy messages. Existing application’s Message composer supports for lengthy messages but it manages a constant value for every user input. While we were developing the application we got a requirement to build a growing message composer.

Final output of this implementation produces a message composer that grows when user completes a new line until user completes 5 lines and after 5 lines it maintains a fixed size and enables scrolling.

So we tried several packages to get this done. And finally we did this  using react-textarea-autosize  it gives all these features and it gives user to customize the elements furthermore.

First we have to install the npm package:

npm install --save react-textarea-autosize

After the installation we have to import the package on top of the “MessageComposer.react.js”

import TextareaAutosize from 'react-textarea-autosize';

Next we need to use this package like this,

         <TextareaAutosize
           className='scroll'
           id='scroll'
           minRows={1}
           maxRows={5}
           placeholder="Type a message..."
           value={this.state.text}
           onChange={this._onChange.bind(this)}
           onKeyDown={this._onKeyDown.bind(this)}
           ref={(textarea) => { this.nameInput = textarea; }}
           style={{ background: this.props.textarea}}
         />

 

This package provides “minRows” and “maxRows”  attributes and we can define minimum height of the text area and maximum height it can grow. If you need to know more about auto growing text areas and to get examples refer this.

Next we wanted to hide the scrollbar which is displaying when the textarea height is exceeding.

How we hide the scrollbars  on chrome browsers.

.scroll::-webkit-scrollbar {
 	 display: none;
}

This is how we hide the scrollbar on firefox browser.

.scroll {
 	overflow: -moz-scrollbars-none;
}

Now we have to style up the textarea because it comes with default styles. We wrapped up the textarea with the div and applied our styles to that. In my case we wrapped up my textarea with  <div className=“textBack”>

This is how we styled the textarea using the wrapper div.

.textBack{
 background: #fff;
 width: 83%;
 border-radius: 40px;
 padding: 5px 20px;
 display: block;
 position: relative;
 top: 12%;
 box-sizing: content-box;
 margin: 0px 0 10px 0;
}

Our textarea is like this.

It expands when user exceeds the width of textarea.

This is how we implemented the SUSI Web Chat’s growing message composer. If you would like to contribute please fork our repository on github  

Resources:

Implementing Hiding App Bar of SUSI Web Chat Application

In the SUSI Web Chat application we got a requirement to build a responsive app bar for static pages and there was another requirement to  show and hide the app bar when user scrolls. Basically this is how it should work: The app bar should be hidden after user scrolls down to a certain extent. When user scrolls up, It should appear again.

First we tried readymade node packages to do this task. But these packages are hard to customize. So we planned to make this feature from the sketch. We used Jquery for this. This is how we built this.

First we installed jQuery package using this command.

npm install jquery

Next we imported it on top of the application like this.

import $ from 'jquery'

We have discussed about this app bar and how we made it in previous blog post. Our app bar is like this.

             <header className="nav-down" id="headerSection">
             <AppBar
               className="topAppBar"
               title={<img src="susi-white.svg" alt="susi-logo"
               className="siteTitle"/>}
               style={{backgroundColor:'#0084ff'}}
               onLeftIconButtonTouchTap={this.handleDrawer}
               iconElementRight={<TopMenu />}
             />
             </header>

We have to use these HTML elements to write jQuery code. But we can’t refer HTML elements before it renders. So we have to define it soon after the render method executes. We can do it using “React LifeCycle” method. We have to add our code into the “componentDidMount()” method.
This is how we used jQuery inside the “componentDidMount()” lifeCycle method. Here we assigned the height of the App Bar using “$(‘header’).outerHeight();”

componentDidMount(){
     var didScroll;
     var lastScrollTop = 0;
     var delta = 5;
     var navbarHeight = $('header').outerHeight();

Here we assigned the height of the app bar to “navbarHeight” variable.

     $(window).scroll(function(event){
         didScroll = true;
     });

In this part we checked whether the user has scrolled or not. If user scrolled we set the value of “didScroll” to “true”.
Now we have to define what to do if user has scrolled.

     function hasScrolled() {
         var st = $(window).scrollTop();
         if(Math.abs(lastScrollTop - st) <= delta){

             return;
         }

Here we get the absolute scrolled height. If the height is less than the delta value we defined, it does not do anything. It just returns.

         if (st > lastScrollTop && st > navbarHeight){
             $('header').removeClass('nav-down').addClass('nav-up');
         } else if(st + $(window).height() < $(document).height()) {
             $('header').removeClass('nav-up').addClass('nav-down');
         }
         lastScrollTop = st;
     }

Here we hide the app bar after user scrolled down more than the height of the app bar. If we need to change the height which app bar should disappear, we just need to add a value to the condition like this.

if (st > lastScrollTop && st > navbarHeight + 200){

If the user scrolled down more than that value we change the class name of the element “nav-down” to “nav-up”.
And we change the className “nav-up” to “nav-down” when user is scrolling up.
We defined CSS classes in the stylesheet to do these things and the animations of action.

header {
   background: #f5b335;
   height: 40px;
   position: fixed;
   top: 0;
   transition: top 0.5s ease-in-out;
   width: 100%;
}

.nav-up {
   top: -100px;
}

We have defined the things which we need to do when user scrolls.
Now we have to call this function if user has scrolled

     setInterval(function() {
         if (didScroll) {
             hasScrolled();
             didScroll = false;
         }
     }, 2500);
   }

If the “didcroll” is “true” we execute the “hasScrolled()” function. And set 2500 millisecond time interval. Because of that app bar does not hide right after user scrolls. It triggers the function after 2.5 seconds later.
This is how we built the scroll bar hiding feature using react JS and jQuery.

Resources:

  • Learn more about React LifeCycle Methods http://busypeoples.github.io/post/react-component-lifecycle/
  • Use jQuery in React component: https:[email protected]/using-jquery-in-react-component-the-refs-way-969de9aa651f

Implementing Speech to Text for Chrome in SUSI Web Chat

SUSI Web Chat now replies to voice inputs. To achieve this, I made use of the Web Speech API. The voice input saves one from the pain of typing and it’s a much needed feature for the Web Chat and to maintain the similarity with the other SUSI Android and SUSI iOS clients.

To test the feature out in SUSI Web Chat, click on the microphone icon beside the text area on chat.susi.ai.

Say the message once the dialog appears, and you will see the message being sent to the Chat List rendered in text.

Let’s achieve the same result following the steps below.

  1. First, initialize the class Voice Recognition with defaults for the Speech Recognition, for that we create a file VoiceRecognition.js
  1. We first initialize the Speech Recognition API with the window object.
  2. We warn the User with a console message if there is no Speech Recognition API available.
  3. If it’s available call the recognition function using the following line

this.recognition = this.createRecognition(SpeechRecognition)

// Initialise the Speech recognition API
const SpeechRecognition = window.SpeechRecognition
      || window.webkitSpeechRecognition
      || window.mozSpeechRecognition
      || window.msSpeechRecognition
      || window.oSpeechRecognition
    // Warn the user if not available otherwise call the createRecognition function
    if (SpeechRecognition != null) {
      this.recognition = this.createRecognition(SpeechRecognition)
    } else {
      console.warn('The current browser does not support the SpeechRecognition API.');
    }
  }
  1. Then we write the createRecognition function
  1. We set our defaults first as “continuous  – true, interimResults – false, and language – ‘en-US’ ”
  2. We pass these options to the recognition object that we created in the above step and finally return the recognition object.
createRecognition = (SpeechRecognition) => {
    const defaults = {
      continuous: true,
      interimResults: false,
      lang: 'en-US'
    }

    const options = Object.assign({}, defaults, this.props)

    let recognition = new SpeechRecognition()

    recognition.continuous = options.continuous
    recognition.interimResults = options.interimResults
    recognition.lang = options.lang

    return recognition
  }
  1. Initialize all the helper functions to be passed as props.
  1. start – This method starts the recognition and invokes the Mic of the browser. It also checks if the browser has the access to the user’s Mic.
  2. stop – Stop method closes the Mic and returns the audio captured so far.
  3. abort – Abort method stops the SpeechRecognition service.
  4. onspeechend – This method is called if there is any inactivity and there is no voice input. Hence, stops the recognition service.
  5. componentWillReceiveProps – This method waits for the stop method and calls it when it has received the stop object.
  6. componentWIllUnmount – This method is invoked just before the component is about to unmount and therefore its function is to abort the Speech Recognition Service
  7. render –  We return null as there is nothing to return in this component and all the converted text of the captured Speech will be sent to the parent element.
start = () => {
    this.recognition.start()
  }

  stop = () => {
    this.recognition.stop()
  }

  abort = () => {
    this.recognition.abort()
  }
  onspeechend = () => {
    console.log('no sound detected');
    this.recognition.stop()
  }

  componentWillReceiveProps ({ stop }) {
    if (stop) {
      this.stop()
    }
  }

  componentWillUnmount () {
    this.abort()
  }

  render () {
    return null
  }
  1. Add event listeners to start and stop functions inside componentDidMount() to ensure every action that we want to perform from the parent element is after the component has successfully mounted itself.
  1. start – The start method is set with an action start so that we can pass the required action name to the VoiceRecognition component that we created
  2. end – The end method similarly is set with an action end
  3. After setting up the actions we finally call the bindResult function with the result that we received.
componentDidMount () {
    const events = [
      { name: 'start', action: this.props.onStart },
      { name: 'end', action: this.props.onEnd },
      { name: 'onspeechend', action: this.props.onspeechend }
    ]

    events.forEach(event => {
      this.recognition.addEventListener(event.name, event.action)
    })

    this.recognition.addEventListener('result', this.bindResult)

    this.start()
  }
  1. Bind the result and send it as the props to the parent element.
  2. Combine all interim results of the recognition and send it to the onResult function as finalTranscript
  3. The function bindResult – The function bindResult does all the binding of the interim results that we received and output a final result as finalTranscript.
  4. Lastly, we add the prop validations to ensure the correct props are being passed to our VoiceRecognition component.
// bindResult function
 bindResult = (event) => {
    let interimTranscript = ''
    let finalTranscript = ''
   // Bind all the results to finalTranscript
    for (let i = event.resultIndex; i < event.results.length; ++i) {
      if (event.results[i].isFinal) {
        finalTranscript += event.results[i][0].transcript
      } else {
        interimTranscript += event.results[i][0].transcript
      }
    }

    this.props.onResult({ interimTranscript, finalTranscript })
  }
// Add Prop Validations
VoiceRecognition.propTypes = {
  onStart: PropTypes.func,
  onEnd : PropTypes.func,
  onResult: PropTypes.func,
  Onspeechend: PropTypes.func,
  continuous: PropTypes.bool,
  lang: PropTypes.string,
  stop: PropTypes.bool
};
// Finally export the VoiceRecognition Component
export default VoiceRecognition
  1. Lastly, call the VoiceRecogntion component and pass the props from the MessageComposer Section to it in the following way.
  1. Initialize the default state in the constructor inside this.state
this.state = {
      text: '',
      start: false, // Starting the VoiceRecognition
      stop: false, // Stop the VoiceRecognition
      open: false, // Maintain the modal state
      result:'' // Maintain the result state
    };
  1. onStart function to call the VoiceRecognition component only when the Mic Button is pressed.
  2. onEnd to end the Speech Recognition service.
  3. onResult to send the message through the Actions.createMessage() function
onResult = ({interimTranscript,finalTranscript }) => {
    let result = interimTranscript;
    let voiceResponse = false;
    this.setState({result:result});
    if(finalTranscript) {
      result = finalTranscript;
      this.setState({
      start: false,
      result:result,
      stop: false,
      open:false,
      animate:false
      });
      if(this.props.speechOutputAlways || this.props.speechOutput){
        voiceResponse = true;
      }
      Actions.createMessage(result, this.props.threadID, voiceResponse);
      setTimeout(()=>this.setState({result: ''}),400);
      this.Button = <Mic />
    }
  }
  1. Fire the component based on the value of start variable and pass the requisite props as given below in the code.
// Only when the start is ‘true’ call the VoiceRecognition component

    {this.state.start && (
          <VoiceRecognition
            onStart={this.onStart}
            onEnd={this.onEnd}
            onResult={this.onResult}
            continuous={true}
            lang="en-US"
            stop={this.state.stop}
          />
        )}

 

  1. Update the text in the “Speak Now” Dialog to show the user the Speech to Text conversion
  1. Update the text in the Modal when it is converted from Speech to Text, i.e. when we set the state of the result variable.
{this.state.result !=='' ? this.state.result :
          'Speak Now...'}

To get access to the full code, go to the repository https://github.com/fossasia/chat.susi.ai

Resources

Implementing Text to Speech on SUSI Web Chat

SUSI Web Chat now gives voice replies while chatting with it similar to SUSI Android and SUSI iOS clients. To test the Voice Output on Chrome,

  1. Visit chat.susi.ai
  2. Click on the Mic input button.
  3. Say something using the Mic when the Speak Now View appears

The simplest way to add text-to-speech to your website is by using the official Speech API currently available for Chrome Browser. The following steps help to achieve it in ReactJS.

  1. Initialize state defaults in a component named, VoicePlayer.
const defaults = {
      text: '',
      volume: 1,
      rate: 1,
      pitch: 1,
      lang: 'en-US'
    } 

There are two states which need to be maintained throughout the component and to be passed as such in our props which will maintain the state. Thus our state is simply

this.state = {
      started: false,
      playing: false
    }
  1. Our next step is to make use of functions of the Web Speech API to carry out the relevant processes.
  1. speak() – window.speechSynthesis.speak(this.speech) – Calls the speak method of the Speech API
  2. cancel() – window.speechSynthesis.cancel() – Calls the cancel method of the Speech API
  1. We then use our component helper functions to assign eventListeners and listen to any changes occurring in the background. For this purpose we make use of the functions componentWillReceiveProps(), shouldComponentUpdate(), componentDidMount(), componentWillUnmount(), and render().
  1. componentWillReceiveProps() – receives the object parameter {pause} to listen to any paused action
  2. shouldComponentUpdate() simply returns false if no updates are to be made in the speech outputs.
  3. componentDidMount() is the master function which listens to the action start, adds the eventListener start and end, and calls the speak() function if the prop play is true.
  4. componentWillUnmount() destroys the speech object and ends the speech.

Here’s a code snippet for Function componentDidMount()

componentDidMount () {
  
    const events = [
      { name: 'start', action: this.props.onStart }
    ]
    // Adding event listeners
    events.forEach(e => {
      this.speech.addEventListener(e.name, e.action)
    })

    this.speech.addEventListener('end', () => {
      this.setState({ started: false })
      this.props.onEnd()
    })

    if (this.props.play) {
      this.speak()
    }
  }
  1. We then add proper props validation in the following way to our VoicePlayer component.
VoicePlayer.propTypes = {
  play: PropTypes.bool,
  text: PropTypes.string,
  onStart: PropTypes.func,
  onEnd: PropTypes.func
};
  1. The next step is to pass the props from a listener view to the VoicePlayer component. Hence the listener here is the component MessageListItem.js from where the voice player is initialized.
  1. First step is to initialise the state.
this.state = {
      play: false,
    }
  onStart = () => {
    this.setState({ play: true });
  }
  onEnd = () => {
    this.setState({ play: false });
  }
  1. Next, we set play to true when we want to pass the props and the text which is to be said and append it to the message lists which have voice set as `true`
 { this.props.message.voice &&
      (<VoicePlayer
             play
             text={voiceOutput}
             onStart={this.onStart}
             onEnd={this.onEnd}
 />)}

Finally, our message lists with voice true will be heard on the speaker as they have been spoken on the microphone. To get access to the full code, go to the repository https://github.com/fossasia/chat.susi.ai or on our chat channel at gitter.im/fossasia/susi_webchat

Resources

  1. Speak-easy-synthesis repository http://mdn.github.io/web-speech-api/speak-easy-synthesis
  2. Web-speech-api repository https://github.com/mdn/web-speech-api/

Showing Offline and Online Status in SUSI Web Chat

A lot of times while chatting on SUSI Web Chat, one does not receive responses, this could either be due to no Internet connection or a down server.

For a better user experience, the user needs to be notified whether he is connected to the SUSI Chat. If one ever loses out on the Internet connection, SUSI Web Chat will notify you what’s the status through a snack bar.

Here are the steps to achieve the above:-

  1. The first step is to initialize the state of the Snackbar and the message.
this.state = {
snackopen: false,
snackMessage: 'It seems you are offline!'
}
  1. Next, we add eventListeners in the component which will bind to our browser’s window object. While this event can be added to any component in the App but we need it in the MessageSection particularly to show the snack bar.

The window object listens to any online offline activity of the browser.

  1. In the file MessageSection.react.js, inside the function componentDidMount()
  1. We initialize window listeners in our constructor section and bind them to the function handleOnline and handleOffline to set states to the opening of the SnackBar.
// handles the Offlines event
window.addEventListener('offline', this.handleOffline.bind(this));  

// handles the Online event
window.addEventListener('online', this.handleOnline.bind(this));
  1. We then create the handleOnline and handleOffline functions which sets our state to make the Snackbar open and close respectively.
// handles the Offlines event
window.addEventListener('offline', this.handleOffline.bind(this));  

// handles the Online event
window.addEventListener('online', this.handleOnline.bind(this));
  1. Next, we create a Snackbar component using the material-ui snackbar component.

The Snackbar is visible as soon as the the snackopen state is set to true with the message which is passed whether offline or online.

<Snackbar
       autoHideDuration={4000}
       open={this.state.snackopen}
       message={this.state.snackMessage}
 />

To get access to the full code, go to the repository https://github.com/fossasia/chat.susi.ai

Resources