Use of Flux Architecture in SUSI.AI

SUSI Web Chat is based on Flux, a React.js Architecture. It provides a simple way for implementing many features in SUSI Web Chat. This blog post explains how Flux Architecture works, and also how it is used in SUSI Web Chat to implement the feature of switching between Light and Dark themes.

What is Flux?

Flux is a micro-architecture for software application development. It is the application architecture that Facebook uses for building client-side web applications. It complements React’s composable view components by utilizing a unidirectional data flow.

Flux Overview

As can be seen from the above diagram, Flux has four main components:

Component Description
Actions Helper methods that pass data to the Dispatcher.
Dispatcher Receives these Actions and broadcast payloads to registered callbacks.
Stores The Stores that are registered to the Dispatcher will update the Views accordingly.
Views  Views are actually React Components that grab the state from the stores and then pass it down to the child components.

How these 4 components work together?

  • Application Initialisation:
    1. Stores let the registered Dispatcher know that they want to be notified whenever an action comes in.
    2. Then the controller views ask the stores for the latest state.
    3. When the stores give the state to the controller views, they pass that state along to their child views to render.
    4. The controller views also ask the stores to keep them notified when state changes so that they can re-render accordingly.
  • The Data Flow:

Once the setup is done, the application is ready to accept user input. So let us trigger an action by having the user make a change.

    1. The view tells the action creator to prepare an action.
    2. The action creator formats the action and sends it off to the Dispatcher.
    3. The Dispatcher dispatches the action off to all of the stores in sequence. Then the store decides whether it cares about this action or not, and changes the state accordingly.
    4. Once it’s done changing state, the store lets its subscribed view controllers know.
    5. Those view controllers will then ask the store to give them the updated state.
    6. After the store gives it the state, the view controller will tell its child views to re-render based on the new state.

So this is how Flux works. Now let us see an example of Flux being used in SUSI Web Chat.

Use of Flux Architecture to Switch Between Themes in SUSI Web Chat

Different Flux components used in SUSI Web Chat:

  1. Actions – Settings.actions.js
  2. Dispatcher – ChatAppDispatcher.js
  3. Store – UserPreferencesStore.js
  4. View – MessageSection.react.js

Let us now see in detail how we are able to switch themes:
In the MessageSection.react.js file, we have the following functions to handle theme change:

handleThemeChanger = () => {
    //code
    this.applyLightTheme();
}

applyLightTheme = () =>{
    //code
    Actions.settingsChanged(settingsChanged);
}

 
Hence, the view tells the action creator to prepare an action.
Now, the settingsChanged() function in Settings.actions.js file is called as follows:

export function settingsChanged(settings) {
    ChatAppDispatcher.dispatch({
    type: ActionTypes.SETTINGS_CHANGED,
    settings
});
    //code
}

 
Hence, the process of invoking the callbacks is initiated through the dispatch() method, which takes a data payload object as its sole argument.

Now, as the UserPreferencesStore is registered with the ChatAppDispatcher, it receives the actions dispatched by the Dispatcher. It checks for the type of action, and with the help of switch cases, executes the code for the corresponding action type accordingly. In this case, the states inside the Store are updated with the new state which the User wants to switch to.

UserPreferencesStore.dispatchToken = ChatAppDispatcher.register(action => {
    switch (action.type) {
        //code
        case ActionTypes.SETTINGS_CHANGED: {
            let settings = action.settings;
            //code
            UserPreferencesStore.emitChange();
            break;
        }
        //code
    }
});

 
This is how Flux Architecture is facilitating the switching of themes in SUSI Web Chat.

Resources

 

Skill Development using SUSI Skill CMS

There are a lot of personal assistants around like Google Assistant, Apple’s Siri, Windows’ Cortana, Amazon’s Alexa, etc. What is then special about SUSI.AI which makes it stand apart from all the different assistants in the world? SUSI is different as it gives users the ability to create their own skills in a Wiki-like system. You don’t need to be a developer to be able to enhance SUSI. And, SUSI is an Open Source personal assistant which can do a lot of incredible stuff for you, made by you.

So, let’s say you want to create your own Skill and add it to the existing SUSI Skills. So, these are the steps you need to follow regarding the same –

  1. The current SUSI Skill Development Environment is based on an Etherpad. An Etherpad is a web-based collaborative real-time editor. https://dream.susi.ai/ is one such Etherpad. Open https://dream.susi.ai/ and name your dream (in lowercase letters).
  2. Define your skill in the Etherpad. The general skill format is

::name <Skill_name>
::author <author_name>
::author_url <author_url>
::description <description> 
::dynamic_content <Yes/No>
::developer_privacy_policy <link>
::image <image_url>
::term_of_use <link>

#Intent
User query1|query2|query3....
Answer answer1|answer2|answer3...

 

Patterns in query can be learned easily via this tutorial.

  1. Open any SUSI Client and then write dream <your dream name> so that dreaming is enabled for SUSI. Once dreaming is enabled, you can now test any skills which you’ve made in your Etherpad.
  2. Once you’ve tested your skill, write ‘stop dreaming’ to disable dreaming for SUSI.
  3. If the testing was successful and you want your skill to be added to SUSI Skills, send a Pull Request to susi_skill_data repository providing your dream name.

How do you modify an existing skill?

SUSI Skill CMS is a web interface where you can modify the skills you’ve made. All the skills of SUSI are directly in sync with the susi_skill_data.

To edit any skill, you need to follow these steps –

  1. Login to SUSI Skill CMS website using your email and password (or Sign Up to the website if you haven’t already).
  2. Click on the skill which you want to edit and then click on the “edit” icon.
  3. You can edit all aspects of the skill in the next state. Below is a preview:

Make the changes and then click on “SAVE” button to save the skill.

What’s happening Behind The Scenes in the EDIT process?

  • SkillEditor.js is the file which is responsible for keeping a check over various validations in the Skill Editing process. There are certain validations that need to be made in the process. Those are as follows –
  • Check whether User has logged in or not

if (!cookies.get('loggedIn')) {
            notification.open({
                message: 'Not logged In',
                description: 'Please login and then try to create/edit a skill',
                icon: <Icon type='close-circle' style={{ color: '#f44336' }} />,
            });
            this.setState({
                loading: false
            });
            return 0;
        }

 

  • Check whether Commit Message has been entered by User or not

if (this.state.commitMessage === null) {
            notification.open({
                message: 'Please add a commit message',
                icon: <Icon type='close-circle' style={{ color: '#f44336' }} />,
            });

            this.setState({
                loading: false
            });
            return 0;
        }

 

  • Check to ensure that request is sent only if there are some differences in old values and new values

if (this.state.oldGroupValue === this.state.groupValue &&
          this.state.oldExpertValue === this.state.expertValue &&
          this.state.oldLanguageValue === this.state.languageValue &&
          !this.state.codeChanged && !this.state.image_name_changed) {
            notification.open({
                message: 'Please make some changes to save the Skill',
                icon: <Icon type='close-circle' style={{ color: '#f44336' }} />,
            });
            self.setState({
                loading: false
            });
            return 0;
        }

 

  • After doing the above validations, a request is sent to the Server and the User is shown a notification accordingly, whether the Skill has been uploaded to the Server or there has been some error.

$.ajax(settings)
            .done(function (response) {
                this.setState({
                    loading: false
                });
                let data = JSON.parse(response);
                if (data.accepted === true) {
                    notification.open({
                        message: 'Accepted',
                        description: 'Your Skill has been uploaded to the server',
                        //success/>
                    });
                }
                else {
                    this.setState({
                        loading: false
                    });
                    notification.open({
                        message: 'Error Processing your Request',
                        description: String(data.message),
                        //failure />
                    });
                }
            }

 

  • If the User is notified with a Success notification, then to verify whether the Skill has been added or not, the User can go to susi_skill_data repo and see if he has a recent commit regarding the same or not.

Resources

Store User’s Personal Information with SUSI

In this blog, I discuss how SUSI.AI stores personal information of it’s users. This personal information is mostly about usernames/links to different websites like LinkedIn, GitHub, Facebook, Google/Gmail etc. To store such details, we have a dedicated API. Endpoint is :

https://api.susi.ai/aaa/storePersonalInfo.json

In this API/Servlet, storing the details and getting the details, both the aspects are covered. At the time of making the API call, user has an option either to ask the server for a list of available store names along with their values or request the server to store the value for a particular store name. If a store name already exists and a client makes a call with new/updated value, The servlet will update the value for that particular store name.

The reason you are looking at minimal user role as USER is quite obvious, i.e. these details correspond to a particular user. Hence neither we want someone writing such information anonymously nor we want this information to be visible to anonymous user until allowed by the user.

In the next steps, we start evaluating the API call made by the client. We look at the combination of the parameters present in the request. If the request is to fetch list of available stores, server first checks if Accounting object even has a JSONObject for “stores” or not. If not found, it sends an error message “No personal information is added yet.” and error code 420. Prior to all these steps, server first generates an accounting object for the user. If found, details are encoded as JSONObject’s parameter. Look at the code below to understand things fairly.

Accounting accounting = DAO.getAccounting(authorization.getIdentity());
        if(post.get("fetchDetails", false)) {
            if(accounting.getJSON().has("stores")){
                JSONObject jsonObject = accounting.getJSON().getJSONObject("stores");
                json.put("stores", jsonObject);
                json.put("accepted", true);
                json.put("message", "details fetched successfully.");
                return new ServiceResponse(json);
            } else {
                throw new APIException(420, "No personal information is added yet.");
            }
        }

If the request was not to fetch the list of available stores, It means client wants server to save a new field or update a previous value for that of a store name. A combination of If-else evaluates whether the call even contains required parameters.

if (post.get(“storeName”, null) == null) {
throw new APIException(422, “Bad store name encountered!”);
}

String storeName = post.get(“storeName”, null);
if (post.get(“value”, null) == null) {
throw new APIException(422, “Bad store name value encountered!”);
}

If request contains all the required data, then store name & value are extracted as key-value pair from the request.

In the next steps, since the server is expected to store list of the store names for a particular user, First the identity is gathered from the already present authorization object in “serviceImpl” method. If the server finds a “null” identity, It throws an error with error code 400 and error message “Specified User Setting not found, ensure you are logged in”.

Else, server first checks if a JSONObject with key “stores” exists or not. If not, It will create an object and will put the key value pair in the new JSONObject. Otherwise it would anyways do so.

Since these details are for a particular account (i.e. for a particular user), these are placed in the Accounting.json file. For better knowledge, Look at the code snippet below.

if (accounting.getJSON().has("stores")) {
                accounting.getJSON().getJSONObject("stores").put(storeName, value);
            } else {
                JSONObject jsonObject = new JSONObject(true);
                jsonObject.put(storeName, value);
                accounting.getJSON().put("stores", jsonObject);
            }

            json.put("accepted", true);
            json.put("message", "You successfully updated your account information!");
            return new ServiceResponse(json);

Additional Resources :

SUSI.AI User Roles and How to Modify Them

In this blog, I discuss what is ‘user-role’ in SUSI.AI, what are the various roles and how SUSI admins can modify/update a user’s roles.

What is User Role?

A UserRole defines the servlet access right. Not all users are allowed to access all the data and services. For  example, To list all the users, minimal user role expected is ADMIN. This classification of users are inspired by the wikipedia User Access Levels, see https://en.wikipedia.org/wiki/Wikipedia:User_access_levels.While querying SUSI, Users are classified into 7 different categories, namely :

  • BOT
  • ANONYMOUS
  • USER  
  • REVIEWER
  • ACCOUNTCREATOR
  • ADMIN
  • BUREAUCRAT

* Please see that these are as of the date of publish of this blog. These are subject to change, which is very unlikely.

If SUSI is active as a bot on some bot integrated platform (like line or kik), the user role assigned to it will be that of BOT. This user role just has technical access to the server.

All the users who are not logged in but interacting with SUSI are ANONYMOUS users. These are only subject to chat, login and signup. They may use forgot password service and reset password services as well.

Once a user login to the server, a token is generated and sent back to client to maintain the identity, hence acknowledging them as USER(s).

Users with role assigned as “REVIEWERS” are expected to manage the Skill CMS. There might be some dispute or conflict in a skill. REVIEWERS then take the access of skill data and finalise the conflict there itself for smooth functionality.

ADMIN users are those who have special rights with them. These are more like moderators with much special rights than any other user.

At the top level of the hierarchy are the BUREAUCRATS. These users have more rights than anyone. They can change role of any other user, override decision of any ADMIN user as well. Both admins and bureaucrats have the access to all the settings file on the server. They not only can look at the list, but also download and upload them. Now these users also have right to upgrade or downgrade any other user as well.

All these user roles are defined in UserRole.java file.

In each request received by the server, the user role of user making the request is compared with the minimal user role in getMinimalUserRole() method. This method is defined in AbstractAPIHandler which validates if a user is allowed to access a particular servlet or not.

private void process(HttpServletRequest request, HttpServletResponse response, Query query) throws ServletException, IOException {
	// object initialisation and comparsions
// user authorization: we use the identification of the user to get the assigned authorization
        Authorization authorization = DAO.getAuthorization(identity);

        if (authorization.getUserRole().ordinal() < minimalUserRole.ordinal()) {
        	response.sendError(401, "Base user role not sufficient. Your base user role is '" + authorization.getUserRole().name() + "', your user role is '" + authorization.getUserRole().getName() + "'");
			return;
        }
// evaluations based on other request parameters.
}

Now that we know about what User Roles actually are, let us look at how the servlet which allows the users {with at least ADMIN login} to change user role of some other user works.

In the request, 2 parameters are expected. These are :

  • user : email id of the user whose role has to be changed.
  • role : new role which will be assigned to this user.

Using a switch case, we identify the user role which is requested. If role is found to be null or any other value apart from “bot”, “anonymous”, “user”, “reviewer”, “accountcreator”, “admin” or “bureaucrat”, an error with error code 400 and error message “Bad User role” is thrown.

In the next steps, server generates client identity in order to get the corresponding Authorization object. If the user is not found in the database, again an error is thrown with error code 400 and error message “role not found

ClientCredential credential = new ClientCredential(ClientCredential.Type.passwd_login, userTobeUpgraded);
            ClientIdentity identity = new ClientIdentity(ClientIdentity.Type.email, credential.getName());
            if (!DAO.hasAuthorization(identity)) {
                throw new APIException(400, "Username not found");
            }

By now, server is clear with the user identity and new role to be assigned. Since the user role is defined in authorization.json file, we overwrite the existing user role and finally server sends back the new user role of the use

Authorization auth = DAO.getAuthorization(identity);
            try {
                auth.setUserRole(userRole);
            } catch (IllegalArgumentException e) {
                throw new APIException(400, "role not found");
            }

            // Print Response
            result.put("newDetails", auth.getJSON());
            result.put("accepted", true);
            result.put("message", "User role changed successfully!!");
            return new ServiceResponse(result);

 

Making Skill Display Cards Identical in SUSI.AI Skill CMS

SUSI.AI Skill CMS shows all the skills of SUSI.AI. The cards used to display all the skills follow flexbox structure and adjust their height according to content. This lead to cards of different sizes and this needed to be fixed. This needed to fix as the cards looked like this:

The cards display following things:

  • Image related to skill
  • An example query related to skill in double quotes
  • Name of skill
  • Short description of skill

Now to get all these, we make an ajax call to the following endpoint:

http://api.susi.ai/cms/getSkillList.json?model='+ this.state.modelValue + '&group=' + this.state.groupValue + '&language=' + this.state.languageValue

Explanation:

  • this.state.modelValue: This is the model of the skill, stored in state of component
  • this.state.groupValue: This represents the group to which skill belongs to. For example Knowledge, Communication, Music, and Audio, etc.
  • this.state.languageValue: This represents the ISO language code of language in which skill is defined

Now the response is in JSONP format and it looks like:

Now we parse the response to get the information needed and return the following Card(Material UI Component):

<Link key={el}
     to={{
        pathname: '/' + self.state.groupValue + '/' + el + '/' + self.state.languageValue,
            state: {
                        url: url,
                        element: el,
                        name: el,
                        modelValue: self.state.modelValue,
                        groupValue: self.state.groupValue,
                        languageValue: self.state.languageValue,
                       }
           }}>
           <Card style={styles.row} key={el}>
                <div style={styles.right} key={el}>
                       {image ? <div style={styles.imageContainer}>
                        <img alt={skill_name}
                          src={image}
                          style={styles.image} />
                          </div> :
                         <CircleImage name={el} size='48' />}
                             <div style={styles.titleStyle}>{examples}</div>
                             </div>
                             <div style={styles.details}>
                                 <h3 style={styles.name}>{skill_name}</h3>
                                 <p style={styles.description}>{description}</p>
                             </div>
         </Card>
</Link>

Now the information that leads to non-uniformity in these cards is the skill description. Now to solve this we decided to put a certain limit to the description length and if that limit is crossed, then we will show the following dots: “”. The height and width of the cards were fixed according to screen size and we modified the description as follows:

if (skill.descriptions) {
      if (skill.descriptions.length > 120) {
          description = skill.descriptions.substring(0, 119) + '...';
      }
      else {
          description = skill.descriptions;
      }
}

This way no content was being cut and all the skill cards looks identical:

Resources:

Implementing Author’s Skill Page in SUSI.AI CMS

SUSI.AI Skill CMS is improving every day and we keep adding new features to it. Recently a feature was added to display all the skills by an author. This feature only showed the list of skills. The user might want to visit the skill page to see the description so we linked the skills on the list to skill page. The list looked like this:

We need to link skill name and image to respective skill page. Now since this is react based app, we do not have different URL for different skills due to SPA. The description, images and other relevant details of skills were being passed as props. We needed to have routes through which we can directly access the skill. This was done by implementing child routes for Skill CMS. Earlier the description, images, and other relevant data was being passed as props from the BrowseSkill component, but now we need to derive this from the URL:

let baseUrl = 'http://api.susi.ai/cms/getSkillMetadata.json';           
let modelValue = "general";
this.name = this.props.location.pathname.split('/')[2];
this.groupValue = this.props.location.pathname.split('/')[1];
this.languageValue = this.props.location.pathname.split('/')[3];
url = baseUrl + '?model=' + modelValue + '&group=' + this.groupValue +        '&language=' + this.languageValue + '&skill=' + this.name;

We now make an ajax call to this URL for fetching the data:

$.ajax({
               url: url,
               jsonpCallback: 'pc',
               dataType: 'jsonp',
               jsonp: 'callback',
               crossDomain: true,
               success: function (data) {
                   self.updateData(data.skill_metadata)
               }
           });

This updates the skill page with the description, image, author and other relevant details of the skills. Now all left to do is link the skills on the list to their respective links. This is done by following code:

We define skillURL as:

let skillURL = 'http://skills.susi.ai/' + parse[6] + '/' + parse[8].split('.')[0] + '/' + parse[7];

Here parse is an array which contains model, group and ISO language code of the skill. We updated the Image and text component as:

<a
   href={skillURL} >
   <Img
     style={imageStyle}
     src={[
           image1,
           image2
          ]}
     unloader={<CircleImage name={name} size="40"/>}
                          />
</a>
<a
   href={skillURL}
   className="effect-underline" >
    {name}
</a>

Now after proper styling, we had the following looking skill list by author:

Resources

Link Preview Service from SUSI Server

 SUSI Webchat, SUSI Android app, SUSI iOS app are various SUSI clients which depend on response from SUSI Server. The most common response of SUSI Server is in form of links. Clients usually need to show the preview of the links to the user. This preview may include featured image, description and the title of the link.  Clients show this information by using various 3rd party APIs and libraries. We planned to create an API endpoint for this on SUSI Server to give the preview of the link. This service is called LinkPreviewService.
String url = post.get("url", "");
        if(url==null || url.isEmpty()){
            jsonObject.put("message","URL Not given");
            jsonObject.put("accepted",false);
            return new ServiceResponse(jsonObject);
        }

This API Endpoint accept only 1 get parameter which is the URL whose preview is to be shown.

Here we also check if no parameter or wrong URL parameter was sent. If that was the the case then we return an error message to the user.

 SourceContent sourceContent =     TextCrawler.scrape(url,3);
        if (sourceContent.getImages() != null) jsonObject.put("image", sourceContent.getImages().get(0));
        if (sourceContent.getDescription() != null) jsonObject.put("descriptionShort", sourceContent.getDescription());
        if(sourceContent.getTitle()!=null)jsonObject.put("title", sourceContent.getTitle());
        jsonObject.put("accepted",true);
        return new ServiceResponse(jsonObject);
    }

The TextCrawler function accept two parameters. One is the url of the website which is to be scraped for the preview data and the other is depth. To get the images, description and title there are methods built in. Here we just call those methods and set them in our JSON Object.

 private String htmlDecode(String content) {
        return Jsoup.parse(content).text();
    }

Text Crawler is based on Jsoup. Jsoup is a java library that is used to scrape HTML pages.

To get anything from Jsoup we need to decode the content of HTML to Text.

public List<String> getImages(Document document, int imageQuantity) {
        Elements media = document.select("[src]");
        while(var5.hasNext()) {
            Element srcElement = (Element)var5.next();
            if(srcElement.tagName().equals("img")) {
                ((List)matches).add(srcElement.attr("abs:src"));
            }
        }

 The getImages method takes the HTML document from the JSoup and find the image tags in that. We have given the imageQuantity parameter in the function, so accordingly it returns the src attribute of the first n images it find.

This API Endpoint can be seen working on

http://127.0.0.1:4000/susi/linkPreview.json?url=<ANY URL>

A real working example of this endpoint would be http://api.susi.ai/susi/linkPreview.json?url=https://techcrunch.com/2017/07/23/dear-tech-dudes-stop-being-such-idiots-about-women/

Resources:

Web Crawlers: https://www.promptcloud.com/data-scraping-vs-data-crawling/

JSoup: https://jsoup.org/

JSoup Api Docs: https://jsoup.org/apidocs/

Parsing HTML with JSoup: http://www.baeldung.com/java-with-jsoup

Implementing Text To Speech Settings in SUSI WebChat

SUSI Web Chat has Text to Speech (TTS) Feature where it gives voice replies for user queries. The Text to Speech functionality was added using Speech Synthesis Feature of the Web Speech API. The Text to Speech Settings were added to customise the speech output by controlling features like :

  1. Language
  2. Rate
  3. Pitch

Let us visit SUSI Web Chat and try it out.

First, ensure that the settings have SpeechOutput or SpeechOutputAlways enabled. Then click on the Mic button and ask a query. SUSI responds to your query with a voice reply.

To control the Speech Output, visit Text To Speech Settings in the /settings route.

First, let us look at the language settings. The drop down list for Language is populated when the app is initialised. speechSynthesis.onvoiceschanged function is triggered when the app loads initially. There we call speechSynthesis.getVoices() to get the voice list of all the languages currently supported by that particular browser. We store this in MessageStore using ActionTypes.INIT_TTS_VOICES action type.

window.speechSynthesis.onvoiceschanged = function () {
  if (!MessageStore.getTTSInitStatus()) {
    var speechSynthesisVoices = speechSynthesis.getVoices();
    Actions.getTTSLangText(speechSynthesisVoices);
    Actions.initialiseTTSVoices(speechSynthesisVoices);
  }
};

We also get the translated text for every language present in the voice list for the text – `This is an example of speech synthesis` using google translate API. This is called initially for all the languages and is stored as translatedText attribute in the voice list for each element. This is later used when the user wants to listen to an example of speech output for a selected language, rate and pitch.

https://translate.googleapis.com/translate_a/single?client=gtx&sl=en-US&tl=TARGET_LANGUAGE_CODE&dt=t&q=TEXT_TO_BE_TRANSLATED

When the user visits the Text To Speech Settings, then the voice list stored in the MessageStore is retrieved and the drop down menu for Language is populated. The default language is fetched from UserPreferencesStore and the default language is accordingly highlighted in the dropdown. The list is parsed and populated as a drop down using populateVoiceList() function.

let voiceMenu = voices.map((voice,index) => {
  if(voice.translatedText === null){
    voice.translatedText = this.speechSynthesisExample;
  }
  langCodes.push(voice.lang);
  return(
    <MenuItem value={voice.lang}
              key={index}
              primaryText={voice.name+' ('+voice.lang+')'} />
  );
});

The language selected using this dropdown is only used as the language for the speech output when the server doesn’t specify the language in its response and the browser language is undefined. We then create sliders using Material UI for adjusting speech rate and pitch.

<h4 style={{'marginBottom':'0px'}}><Translate text="Speech Rate"/></h4>
<Slider
  min={0.5}
  max={2}
  value={this.state.rate}
  onChange={this.handleRate} />

The range for the sliders is :

  • Rate : 0.5 – 2
  • Pitch : 0 – 2

The default value for both rate and pitch is 1. We create a controlled slider saving the values in state and using onChange function to record change in values. The Reset buttons can be used to reset the rate and pitch values respectively to their default values. Once the language, rate and pitch values have been selected we can click on `Play a short demonstration of speech synthesis`  to listen to a voice reply with the chosen settings.

{ this.state.playExample &&
  (
    <VoicePlayer
       play={this.state.play}
       text={voiceOutput.voiceText}
       rate={this.state.rate}
       pitch={this.state.pitch}
       lang={this.state.ttsLanguage}
       onStart={this.onStart}
       onEnd={this.onEnd}
    />
  )
}

We use the VoicePlayer by passing the required props to get the speech output. onStart and onEnd functions are triggered at the beginning and ending of the speech synthesis and are used to control the state from the parent component. Chosen language, rate, pitch and translated text are passed as props to VoicePlayer which creates a new SpeechSynthesisUtterance() with the passed props and plays the speech output.

On saving these settings and then using the Mic button to get voice replies we see that the voice output is controlled according to the selected settings.

Finally, we have to store the selected settings on the server and ensure that these are pulled when the app is initialized. The format in which these settings are stored in the server is :

Speech Rate

- Used to control rate of speech output.
- SETTING_NAME :  `speechRate`
- SETTING_VALUE : `0.5 - 2`
- DEFAULT_VALUE : `1`
 
Speech Pitch

- Used to control pitch of speech output.
- SETTING_NAME :  `speechPitch`
- SETTING_VALUE : `0 - 2`
- DEFAULT_VALUE : `1`
 
TTS Language

- Used to set the language for Text-To-Speech used when the response from server doesnt specify language and the browser language is also undefined.
- SETTING_NAME :  `ttsLanguage`
- SETTING_VALUE : `Language Code (string)`
- DEFAULT_VALUE : `en-US`

This is how the Text To Speech Settings were implemented in SUSI Web Chat. The complete code can be found at SUSI Web Chat Repository.

PS: To test whether your browser supports Text To Speech, open your browser console and try the following :

  • var msg = new SpeechSynthesisUtterance(‘Hello World’);
  • window.speechSynthesis.speak(msg)

If you get a speech output then the Web API Speech Synthesis is supported by your browser and Text To Speech features of SUSI Web Chat will work. The Web Speech API has support for all latest Chrome browsers as mentioned in the Web Speech API Mozilla docs.However there are few bugs with some Chromium versions please check out more on how to fix them locally here in this link.

Resources: