Connecting SUSI iOS App to SUSI Smart Speaker

SUSI Smart Speaker is an Open Source speaker with many exciting features. The user needs an Android or iOS device to set up the speaker. You can refer this post for initial connection to SUSI Smart Speaker. In this post, we will see how a user can connect SUSI Smart Speaker to iOS devices (iPhone/iPad).

Implementation –

The first step is to detect whether an iOS device connects to SUSI.AI hotspot or not. For this, we match the currently connected wifi SSID with SUSI.AI hotspot SSID. If it matches, we show the connected device in Device Activity to proceed further with setups.

Choosing Room –

Room name is basically the location of your SUSI Smart Speaker in the home. You may have multiple SUSI Smart Speaker in different rooms, so the purpose of adding the room is to differentiate between them.

When the user clicks on Wi-Fi displayed cell, it starts the initial setups. We are using didSelectRowAt method of UITableViewDelegate to get which cell is selected. On clicking the displayed Wi-Fi cell, a popup is open with a Room Location Text field.

override func tableView(_ tableView: UITableView, didSelectRowAt indexPath: IndexPath) {
if indexPath.row == 0, let speakerSSID = fetchSSIDInfo(), speakerSSID == ControllerConstants.DeviceActivity.susiSSID {
// Open a popup to select Rooms
presentRoomsPopup()
}
}

When the user clicks the Next button, we send the speaker room location to the local server of the speaker by the following API endpoint with room name as a parameter:

http://10.0.0.1:5000/speaker_config/

Refer this post for getting more detail about how choosing room work and how it is implemented in SUSI iOS.

Sharing Wi-Fi Credentials –

On successfully choosing the room, we present a popup that asks the user to enter the Wi-Fi credentials of previously connected Wi-Fi so that we can connect our Smart Speaker to the wifi which can provide internet connection to play music and set commands over the speaker.

We present a popup with a text field for entering wifi password.

When the user clicks the Next button, we share the wifi credentials to wifi by the following API endpoint:

http://10.0.0.1:5000/wifi_credentials/

With the following params-

  1. Wifissid – Connected Wi-Fi SSID
  2. Wifipassd – Connected Wi-Fi password

In this API endpoint, we are sharing wifi SSID and wifi password with Smart Speaker. If the credentials successfully accepted by speaker than we present a popup for user SUSI account password, otherwise we again present Enter Wifi Credentials popup.

Client.sharedInstance.sendWifiCredentials(params) { (success, message) in
DispatchQueue.main.async {
self.alertController.dismiss(animated: true, completion: nil)
if success {
self.presentUserPasswordPopup()
} else {
self.view.makeToast("", point: self.view.center, title: message, image: nil, completion: { didTap in
UIApplication.shared.endIgnoringInteractionEvents()
self.presentWifiCredentialsPopup()
})
}
}
}

 

Sharing SUSI Account Credentials –

In the method above we have seen that when SUSI Smart Speaker accept the wifi credentials, we proceed further with SUSI account credentials. We open a popup to Enter user’s SUSI account password:

When the user clicks the Next button, we use following API endpoint to share user’s SUSI account credentials to SUSI Smart Speaker:

http://10.0.0.1:5000/auth/

With the following params-

  1. email
  2. password

User email is already saved in the device so the user doesn’t have to type it again. If the user credentials successfully accepted by speaker then we proceed with configuration process otherwise we open up Enter Password popup again.

Client.sharedInstance.sendAuthCredentials(params) { (success, message) in
DispatchQueue.main.async {
self.alertController.dismiss(animated: true, completion: nil)
if success {
self.setConfiguration()
} else {
self.view.makeToast("", point: self.view.center, title: message, image: nil, completion: { didTap in
UIApplication.shared.endIgnoringInteractionEvents()
self.presentUserPasswordPopup()
})
}
}
}

 

Setting Configuration –

After successfully sharing SUSI account credentials, following API endpoint is using for setting configuration.

http://10.0.0.1:5000/config/

With the following params-

  1. sst
  2. tts
  3. hotword
  4. wake

The success of this API call makes successfully connection between user iOS Device and SUSI Smart Speaker.

Client.sharedInstance.setConfiguration(params) { (success, message) in
DispatchQueue.main.async {
if success {
// Successfully Configured
self.isSetupDone = true
self.view.makeToast(ControllerConstants.DeviceActivity.doneSetupDetailText)
} else {
self.view.makeToast("", point: self.view.center, title: message, image: nil, completion: { didTap in
UIApplication.shared.endIgnoringInteractionEvents()
})
}
}
}

After successful connection-

 

Resources –

  1. Apple’s Documentation of tableView(_:didSelectRowAt:) API
  2. Initial Setups for Connecting SUSI Smart Speaker with iPhone/iPad
  3. SUSI Linux Link: https://github.com/fossasia/susi_linux
  4. Adding Option to Choose Room for SUSI Smart Speaker in iOS App
Continue Reading

Adding Support for Playing Youtube Videos in SUSI iOS App

SUSI supports very exciting features in chat screen, from simple answer type to complex map, RSS, table etc type responses. Even user can ask SUSI for the image of anything and SUSI response with the image in the chat screen. What if we can play the youtube video from SUSI, we ask SUSI for playing videos and it can play youtube videos, isn’t it be exciting? Yes, SUSI can play youtube videos too. All the SUSI clients (iOS, Android, and Web) support playing youtube videos in chat.

Google provides a Youtube iFrame Player API that can be used to play videos inside the app only instead of passing an intent and playing the videos in the youtube app. iFrame API provide support for playing youtube videos in iOS applications.

In this post, we will see how playing youtube video features implemented in SUSI iOS.

Getting response from server side –

When we ask SUSI for playing any video, in response, we get youtube Video ID in video_play action type. SUSI iOS make use of Video ID to play youtube video. In response below, you can see that we are getting answer action type and in the expression of answer action type, we get the title of the video.

actions:
[
{
type: "answer",
expression: "Playing Kygo - Firestone (Official Video) ft. Conrad Sewell"
},
{
identifier: "9Sc-ir2UwGU",
identifier_type: "youtube",
type: "video_play"
}
]

Integrating youtube player in the app –

We have a VideoPlayerView that handle all the iFrame API methods and player events with help of YTPlayer HTML file.

When SUSI respond with video_play action, the first step is to register the YouTubePlayerCell and present the cell in collectionView of chat screen.

Registering the Cell –

register(_:forCellWithReuseIdentifier:) method registers a class for use in creating new collection view cells.

collectionView?.register(YouTubePlayerCell.self, forCellWithReuseIdentifier: ControllerConstants.youtubePlayerCell)

 

Presenting the YouTubePlayerCell –

Here we are presenting the cell in chat screen using cellForItemAt method of UICollectionView.

if message.actionType == ActionType.video_play.rawValue {
if let cell = collectionView.dequeueReusableCell(withReuseIdentifier: ControllerConstants.youtubePlayerCell, for: indexPath) as? YouTubePlayerCell {
cell.message = message
cell.delegate = self
return cell
}
}

 

Setting size for cell –

Using sizeForItemAt method of UICollectionView to set the size.

if message.actionType == ActionType.video_play.rawValue {
return CGSize(width: view.frame.width, height: 158)
}

In YouTubePlayerCell, we are displaying the thumbnail of youtube video using UIImageView. Following method is using to get the thumbnail of particular video by using Video ID –

  1. Getting thumbnail image from URL
  2. Setting image to imageView
func downloadThumbnail() {
if let videoID = message?.videoData?.identifier {
let thumbnailURLString = "https://img.youtube.com/vi/\(videoID)/default.jpg"
let thumbnailURL = URL(string: thumbnailURLString)
thumbnailView.kf.setImage(with: thumbnailURL, placeholder: ControllerConstants.Images.placeholder, options: nil, progressBlock: nil, completionHandler: nil)
}
}

We are adding a play button in the center of thumbnail view so that when the user clicks play button, we can present player.

On clicking the Play button, we are presenting the PlayerViewController, which hold all the player setups, by overFullScreen type of modalPresentationStyle.

@objc func playVideo() {
if let videoID = message?.videoData?.identifier {
let playerVC = PlayerViewController(videoID: videoID)
playerVC.modalPresentationStyle = .overFullScreen
delegate?.loadNewScreen(controller: playerVC)
}
}

The methods above present the youtube player with giving Video ID. We are using YouTubePlayerDelegate method to autoplay the video.

func playerReady(_ videoPlayer: YouTubePlayerView) {
videoPlayer.play()
}

The player can be dismissed by tapping on the light black background.

Final Output –

Resources –

  1. Youtube iOS Player API
  2. SUSI API Sample Response for Playing Video
  3. SUSI iOS Link
Continue Reading
Adding Option to Choose Room for SUSI Smart Speaker in iOS App
SAMSUNG CAMERA PICTURES

Adding Option to Choose Room for SUSI Smart Speaker in iOS App

SUSI Smart Speaker is an open source smart speaker that supports many features. The user can use Android or iOS to connect their device with SUSI Smart Speaker. During initial installation, it is asking from use to enter the Room name. Room name is basically the location of your SUSI Smart Speaker in the home. You may have multiple SUSI Smart Speaker in different rooms, so the purpose of adding the room is to differentiate between them. You can find useful instructions for the initial connection between the iOS device and SUSI Smart Speaker here. It this post, we will see how the adding rooms feature implemented for SUSI iOS.

When the user enters into the Device Activity screen, we check if the iOS device is connected to SUSI.AI Wi-Fi hotspot or not. If the device is connected to SUSI Smart Speaker, it shows the Wi-Fi displayed SSID in Device Activity Screen. On clicking the displayed Wi-Fi cell, a popup is open with a Room Location Text field. The user can enter Room location and by clicking the Next button, proceed further with the setup.

In the popup, there is also an option for choosing rooms, where the list of most common room names is displayed and the user can choose room name from the list.

Presenting Room Picker View Controller –

func presentRoomsPicker() {
let storyboard = UIStoryboard(name: "Main", bundle: nil)
if let roomVC = storyboard.instantiateViewController(withIdentifier: "RoomPickerVC") as? RoomPickerController {
roomVC.deviceActivityVC = self
let roomNVC = AppNavigationController(rootViewController: roomVC)
self.present(roomNVC, animated: true)
}
}

Room Picker View Controller is UITableViewController that display the rooms names in table cells. Some of the most common rooms names displayed are:

let rooms: [String] = ["Bedroom", "Kitchen", "Family Room", "Entryway", "Living Room", "Front Yard", "Guest Room", "Dining Room", "Computer Room", "Downstairs", "Front Porch", "Garage", "Hallway", "Driveway"]

 

Presenting Room Cell –

We are using cellForRowAt method to present the cell.

override func tableView(_ tableView: UITableView, cellForRowAt indexPath: IndexPath) -> UITableViewCell {
let cell = tableView.dequeueReusableCell(withIdentifier: "roomsCell", for: indexPath)
cell.textLabel?.text = rooms[indexPath.row]
cell.imageView?.image = ControllerConstants.Images.roomsIcon
return cell
}

 

Selecting the room from cell –

When the user clicks on the cell, first willSelectRowAt method use to display the right icon in the accessory view that shows which cell is selected.

if let oldIndex = tableView.indexPathForSelectedRow {
tableView.cellForRow(at: oldIndex)?.accessoryType = .none
}
tableView.cellForRow(at: indexPath)?.accessoryType = .checkmark

We are storing the selected room in the following variable and selecting it by using didSelectRowAt method of UITableView.

var selectedRoom: String?
override func tableView(_ tableView: UITableView, didSelectRowAt indexPath: IndexPath) {
selectedRoom = rooms[indexPath.row]
}

In Room Picker Screen, the user has two option, Cancel and Done. If the user clicks the Cancel, we dismiss the Room Picker screen and display the popup with the empty room location text field and with Choose Room option. If the user clicks the Done button, we store the picked room in UserDefaults shared instance and dismiss Room Picker screen with a different popup which has already filled room location in the text field and without choose room option in the popup as in the image below. By clicking the next, the user proceeds with the further setup.

Resources –

  1. Apple’s Documentations on UserDefaults.
  2. Initial Setups for Connecting SUSI Smart Speaker with iPhone/iPad
  3. Apple’s Documentations on tableView(_:cellForRowAt:)
  4. Apple’s Documentations on tableView(_:willSelectRowAt:)
  5. Apple’s Documentations on tableView(_:didSelectRowAt:)
Continue Reading

Adding Support for Playing Audio in SUSI iOS App

SUSI.AI supports various actions like the answer, map, table, video play and many more. You can play youtube videos in the chat screen. It also supports for playing audio in the chat screen. In this post, we will see that how playing audio feature implemented in SUSI iOS.

Getting audio action from server side –

In the chat screen, when we ask SUSI to play audio, we get the audio source from the server side. For example, if we ask SUSI “open the pod bay door”, we get the following action object:

"actions": [
{
"type": "audio_play",
"identifier_type": "youtube",
"identifier": "7qnd-hdmgfk"
},
{
"language": "en",
"type": "answer",
"expression": "I'm sorry, Dave. I'm afraid I can't do that."
}
]

In the above action object, we can see that we get two actions, audio_play and answer. In audio_play action, we are getting an identifier type which tells us about the source of audio. Identifier type can be youtube or local or any other source. When the identifier is youtube, we play audio from youtube stream. In identifier, we get the audio file path. In case of youtube identifier type, we get youtube video ID and play from youtube stream. In answer action type, we get the expression which we display in chat screen after thumbnail.

Implementing Audio Support in App –

We use Google’s youtube Iframe API to stream audio from youtube videos. We have a VideoPlayerView that handle all the iFrame API methods and player events with help of YTPlayer HTML file.

Presenting the YouTubePlayerCell –

If the action type is audio_play, we are presenting the cell in chat screen using cellForItemAt method of UICollectionView.

if message.actionType == ActionType.audio_play.rawValue {
if let cell = collectionView.dequeueReusableCell(withReuseIdentifier: ControllerConstants.youtubePlayerCell, for: indexPath) as? YouTubePlayerCell {
cell.message = message
cell.delegate = self
return cell
}
}

Setting size for cell –

Using sizeForItemAt method of UICollectionView to set the size.

if message.actionType == ActionType.audio_play.rawValue {
return CGSize(width: view.frame.width, height: 158)
}

In YouTubePlayerCell, we fetch thumbnail and display in the cell with a play button. On clicking the play button, we open the player and stream music.

Final Output –

Resources –

  1. Apple’s Documentations on sizeForItemAt
  2. SUSI API Sample for Audio Play Action
  3. YouTube iFrame API for iOS
  4. Apple’s Documentations on cellForItemAt
Continue Reading

Integrating Gravatar and Anonymizing Email Address in Feedback Section in SUSI.AI iOS App

SUSI skills are having a very nice feedback system that allows the user to rate skills from 1-star to 5-star and showing ratings in skills screens. SUSI also allow the user to post feedback about skills and display them. You can check out how posting feedback implemented here and how displaying feedback feature implemented here. To enhance the user experience, we are adding user gravatar in the feedback section and to respect user privacy, we are anonymizing the user email displayed in the feedback section. In this post, we will see how these features implemented in SUSI iOS.

Integrating Gravatar –

We are showing gravatar of the user before feedback. Gravatar is a service for providing globally-unique avatars. We are using user email address to get the gravatar. The most basic gravatar image request URL looks like this:

https://www.gravatar.com/avatar/HASH

where HASH is replaced with the calculated hash for the specific email address we are requesting. We are using the MD5 hash function to hash the user’s email address.

The MD5 hashing algorithm is a one-way cryptographic function that accepts a message of any length as input and returns as output a fixed-length digest value to be used for authenticating the original message.

In SUSI iOS, we have MD5Digest.swift file that gives the hash value of email string. We are using the following method to set gravatar:

if let userEmail = feedback?.email {
setGravatar(from: userEmail)
}
func setGravatar(from emailString: String) {
let baseGravatarURL = "https://www.gravatar.com/avatar/"
let emailMD5 = emailString.utf8.md5.rawValue
let imageString = baseGravatarURL + emailMD5 + ".jpg"
let imageURL = URL(string: imageString)
gravatarImageView.kf.setImage(with: imageURL)
}

Anonymizing User’s Email Address –

Before the implementation of this feature, the user’s full email address was displayed in the feedback section and see all review screen. To respect the privacy of the user, we are now only showing user email until the `@` sign.

In Feedback object, we have the email address string that we modify to show until `@` sign by following way:

if let userEmail = feedback?.email, let emailIndex = userEmail.range(of: "@")?.upperBound {
userEmailLabel.text = String(userEmail.prefix(upTo: emailIndex)) + "..."
}

 

Final Output –

Resources –

  1. Post feedback for SUSI Skills in SUSI iOS
  2. Displaying Skills Feedback on SUSI iOS
  3. What is MD5?
Continue Reading

Implementing a skill rating over time graph section in SUSI Skill CMS

In SUSI.AI skill ratings is an invaluable aspect which greatly helps the users to know which skills are performing better than the rest and are more popular than the others. A robust skill rating system for the skills was developed recently which allows the users to rate skills as per their experience and thus data like average rating, total number of ratings is available but there was no provision to see the rating history or how the skills rating has changed over time, this could be an important aspect for users or developers to know what changes to the skill made it less/more popular. An API is developed at the server to retrieve the ratings over time data, we can use these details to render attractive components for a better visual understanding of how the skill is performing and get statistics like how the skill’s rating has changed over time.

About the API

Endpoint : /cms/getRatingsOverTime.json

Parameters

  • model
  • group
  • language
  • skill

After consuming these params the API will return the number of times a skill is called along with

the date on which it is called. We use that data as an input for the line chart component that we want to render. 

Fetching data from the server and storing in the application state

Make an AJAX call to the server to fetch the data from the URL which holds the server endpoint, on successfully receiving the data we do some formatting with the timestamp that comes along the data to make it more convenient to understand and then we call a saveRatingOverTime function which saves the data received from the server to the application state.

let ratingOverTimeUrl = `${urls.API_URL}/cms/getRatingsOverTime.json`;
skillUsageUrl = skillUsageUrl + '?model=' + modelValue + '&group=' + this.groupValue + '&language=' + this.languageValue + '&skill=' + this.name;
// Fetch the skill ratings over time
$.ajax({
 url: ratingOverTimeUrl,
 dataType: 'json',
 crossDomain: true,
 success: function(data) {
        if (data.ratings_over_time) {
         const ratingData = data.ratings_over_time.map(item => {
             return {
               rating: item.rating,
               count: item.count,
               timestamp: parseDate(item.timestamp)
                 .split(' ')
                 .slice(2, 4)
                 .join(' '),
                 };
           });
         self.saveRatingOverTime(ratingData);
        }
 },
 error: function(e) {
        console.log(e);
        self.saveRatingOverTime();
 },
});

 

Save the skill usage details in the component state.

// Save ratings over time data in the component state
saveRatingOverTime = (ratings_over_time = []) => {
 this.setState({
        ratings_over_time,
 });
};

 

Send the received data as props to the Skill Rating component and render it.

<SkillUsageCard skill_usage={this.state.skill_usage} /> 

Implementing the UI

Importing the packages for rendering the chart in the Skill Ratings component.

import { XAxis, YAxis, Tooltip, LineChart, Line, Legend, ResponsiveContainer } from 'recharts';

 

Display a small heading for the section in the ratings card and Render a Responsive container component which will form a parent component for out Chart which will be rendered when the ratings over time data received in the props is not empty.

<div className="sub-title" style={{ alignSelf: 'flex-start' }}>
 Rating over time
</div>
{this.props.ratings_over_time.length ? (
 <div>
        <ResponsiveContainer
         height={300}
         width={
           window.innerWidth < 660
             ? this.state.width
             : this.state.width * 1.5
         }
         debounce={1}
        >
         ...
        </ResponsiveContainer>
 </div>
) : (
 <div>No ratings data over time is present</div>
)}

 

Render a LineChart and supply data from the data prop received from the Skill Listing component, add X-Axis and Y-Axis by supplying corresponding dataKey props depending on the data received, Add a tooltip to describe points on the line chart and a legend which describes the lines, After that we have a Line component which depicts the change in ratings over time on the chart.

<LineChart
 data={this.props.ratings_over_time}
 margin={{
        top: 5,
        right: 30,
        left: 20,
        bottom: 5,
 }}
>
 <XAxis dataKey="timestamp" padding={{ right: 20 }} />
 <YAxis dataKey="rating" />
 <Tooltip wrapperStyle={{ height: '60px' }} />
 <Legend />
 <Line
        name="Average rating"
        type="monotone"
        dataKey="rating"
        stroke="#82ca9d"
        activeDot={{ r: 8 }}
 />
</LineChart>

 

So I hope after going through this blog it is more clear how the ratings over time section is implemented in the Skill CMS.

Resources

 

Continue Reading

Integrate prettier with lint-staged and ESLint for consistent code style throughout the project

SUSI Skill CMS presently use ESLint to check for code linting errors, the ESLint rules are written in a separate .eslintrc file which lives at the project root. The project didn’t follow any best practices for react apps and the rules were weak therefore a lot of bad/unindented code was present in the project which takes a lot of time to fix manually, not to mention there was no mechanism to auto-format the code while committing. Also, code reviews take a lot of time to discuss code styles and fixing them.

Prettier comes to the rescue as it’s a code formatter which provides a ton of options to achieve the desired well-formatted code. Prettier enforces a consistent code style across your entire codebase because it disregards the original styling by parsing it away and re-printing the parsed code with its own rules

Add prettier as a development dependency

npm install prettier --save-dev --save-exact

 

Similar to how we write ESLint rules in a separate .eslintrc file, we have a .prettierrc file which contains rules for prettier but since we already have ESLint so we run prettier on top of ESLint to leverage functionalities of both packages, this is achieved by using eslint-plugin-prettier and eslint-config-prettier which exist for ESLint. These packages are saved as devDependencies in the project and “prettier” as a plugin is added to .eslintrc file and recommended prettier rules are extended by adding “prettier” in .eslintrc file.

Install the config and plugin packages

npm i eslint-plugin-prettier eslint-config-prettier --save-dev

 

To run prettier using ESLint, add the prettier to ESLint plugins and add prettier errors to ESLint rules.

// .eslintrc
{
 "plugins": ["prettier"],
 "rules": {
   "prettier/prettier": "error"
 }
}

 

Extending Prettier rules in ESLint

// .eslintrc
{
 ...
 "extends": ["prettier"]
 ...
}

 

5856 linting errors found which were undetected initially (SUSI Skill CMS).

image

Add a .prettierrc file with some basic formatting rules for now like enabling single quotes wherever applicable and to have trailing comma at the end of JSON objects.

// .prettierrc
{
   "singleQuote": true,
   "trailingComma": "all",
   "parser": "flow"
}

 

Add a format script in package.json so that user can format the code whenever required manually too.

// package.json
"scripts": {
        ...
        "format": "prettier --write \"src/**/*.js\"",
        ...
},

 

Since we want to prevent contributors from committing unindented code we used lint-staged, a package which runs tasks on a set of specified files. We achieve this by adding a set of tasks in the lint-staged object and then run lint-staged as a pre-commit hook using husky.

Install lint-staged as a development dependency which will be visible in package.json file.

npm i lint-staged --save-dev 

 

Add tasks for lint-staged as an object in package.json, we add a lint-staged object in the pacage.json file and grab all .js files in the project and run eslint, prettier over them and then we finally run git add to stage the changes done by prettier.

// package.json

"lint-staged": {
   "src/**/*.js": [
     "eslint src --ext .js",
     "prettier --write",
     "git add"
   ]
 }

 

Call lint-staged in pre-commit git-hook to run the specified tasks in the package.json.

// package.json

"scripts": {
   ...
   "precommit": "lint-staged",
   ...
},

 

Running lint-staged tasks before committing

image

As a result, we save a lot of time in reviewing the code since we don’t have to be worried about the code styles anymore as pre-commit hook already takes care of that also this ensures consistent code style throughout the project which improves the overall quality.

Resources

Prettier library website
Use ESLint to run Prettier
Link to my Pull Request
eslint-plugin-prettier
eslint-config-prettier

Continue Reading

Display Skill Usage of the Past Week in a Line Chart

Skill usage statistics in SUSI.AI is an important aspect which greatly helps the skill developers know what is more engaging for the users and the users know which skills are more popular than the others and are being used widely. So data like this should be interactively available on the clients. An API is developed at the server to retrieve the skill usage details, we can use these details to render attractive components for a better visual understanding of how the skill is performing and get statistics like on which days the skill has been active in particular.

 

About the API

Endpoint : /cms/getSkillUsage.json

Parameters

  • model
  • group
  • language
  • skill

After consuming these params the API will return the number of times a skill is called along with the date on which it is called. We use that data as an input for the line chart component that we want to render.

Creating a Skill Usage card component

Import the required packages from corresponding libraries. We are using recharts as the library for the charts support and thus we import several components required for the line chart at the top of the Skill Usage component.

// Packages
import React, { Component } from 'react';
import PropTypes from 'prop-types';
import { LineChart, Line, XAxis, YAxis, Tooltip, Legend } from 'recharts';
import { Paper } from 'material-ui';

 

Create a SkillUsageCard component which is enclosed in a Paper component from material ui library to give it a card like look and then we render a Line chart inside it using appropriate data props which are received from the skill page component, we set a height and a width for the chart, Add an X-Axis, Y-Axis, Tooltip which shows up on hovering over points on the graph, legends to describe the line on the chart and then finally a line with several styling and other props.

class SkillUsageCard extends Component {
 render() {
   return(
     <Paper className="margin-b-md margin-t-md">
       <h1 className='title'>
           Skill Usage
       </h1>
       <div className="skill-usage-graph">
         <LineChart width={600} height={300} data={this.props.skill_usage}
               margin={{top: 5, right: 30, left: 20, bottom: 5}}>
           <XAxis dataKey="date" padding={{right: 20}} />
           <YAxis/>
           <Tooltip wrapperStyle={{height: '60px'}}/>
           <Legend />
           <Line name='Skill usage count' type="monotone" dataKey="count" stroke="#82ca9d" activeDot={{r: 8}}/>
         </LineChart>
       </div>
     </Paper>
   )
 }
}

 

Add prop validation at the end of the file to validate the coming props from the skill page component to validate that correct props are being received.

SkillUsageCard.propTypes = {
   skill_usage: PropTypes.array
}

 

Export the component so it can be used in other components.

export default SkillUsageCard;

 

Fetch the data for the component from the API in the skill page component where the skill usage component will be rendered. First set the API url and then make an AJAX call to that URL and once the data is received from the server pass that received data to a saveSkillUsage function which does the simple task of saving the data to the state and passing the saved data as a prop to the skill usage component. In case the call fails we log the error to the console.

let skillRatingUrl = `${urls.API_URL}/cms/getSkillRating.json`;
skillUsageUrl = skillUsageUrl + '?model=' + modelValue + '&group=' + this.groupValue + '&language=' + this.languageValue + '&skill=' + this.name;
// Fetch skill usage of the visited skill
$.ajax({
  url: skillUsageUrl,
  dataType: 'json',
  crossDomain: true,
  success: function (data) {
      self.saveSkillUsage(data.skill_usage)
  },
  error: function(e) {
       console.log(e)
  }
});

 

Save the skill usage details in the component state to render the skill usage component.

saveSkillUsage = (skill_usage = []) => {
  this.setState({
     skill_usage
 })
}

 

Send the received data as props to the Skill Usage component and render it.

<SkillUsageCard skill_usage={this.state.skill_usage} /> 

 

So I hope after reading this blog you have a more clearer insight into how the skill usage details are implemented in the CMS.

Resources –

  • Jerry J. Muzsik, Creating and deploying a react app using recharts URL.
  • Recharts, Documentation, URL.
Continue Reading

Rendering a Uniform StaticAppBar Component across all SUSI Web Clients on all Routes.

The Problem –
We have three SUSI Web Clients namely

Webchat
Skills CMS
Accounts

And it’s important to keep the design guidelines in sync across all the clients, StaticAppBar is a component which forms the header of all the pages and thus it is important to keep it uniform in all clients which was clearly missing before. There is also a lot of code duplication of the AppBar component (in accounts app) since it is used on all the pages so our approach will be to prepare a single component and render it on all routes.

Tackling the problem – Since the StaticAppBar component is present on all the clients we simply make the menu items uniform across all the clients and apply a check on those menu items on which are a premium feature or should appear only once the user is logged in.

Building blocks of the StaticAppBar component

  • AppBar
  • SUSI logo on the left end
  • Drop down hamburger menu on the right

Here’s how the JSX for the StaticAppBar component in CMS looks like, it uses an AppBar component from the material-ui library and has several props and styling as per the requirement.

<AppBar
   className='topAppBar'
   id='appBar'
   title={<div id='rightIconButton' ><Link to='/' style={{ float: 'left', marginTop: '-10px',height:'25px',width:'122px' }}>
       <img src={susiWhite} alt='susi-logo' className='siteTitle' /></Link></div>}
   style={{
       backgroundColor: colors.header,
       height: '46px',
       boxShadow: 'none',
       margin: '0 auto',
   }}
   iconStyleRight={{ marginTop: '-2px' }}
   iconElementRight={<TopRightMenu />}
/>

 

TopRightMenu is a function that returns JSX for the hamburger dropdown and is rendered in the AppBar as depicted below. It is a conditional menubar meaning some menu items are only rendered when the user is logged in and thus this helps cover those features which should only be available to logged in users. After that we use a popover component which shows up when the 3 dots or the expander on the top right is clicked. Almost all of the components in material ui has a style prop so styling is easy for the components moreover the workflow for the popover click goes like once the expander is clicked a boolean state variable named showOptions is toggled which in turn toggles the opening or closing state of the Popover as per the open prop.

let TopRightMenu = (props) => (
           <div onScroll={this.handleScroll}>
               <div>
                   {cookies.get('loggedIn') ?
                       (<label
                           style={{color: 'white', fontSize: '16px', verticalAlign:'super'}}>
                           {cookies.get('emailId')}
                           </label>) :
                       (<label>
                           </label>)
                   }
                   <IconMenu
                       {...props}
                       iconButtonElement={
                           <IconButton
                               iconStyle={{ fill: 'white' }}><MoreVertIcon /></IconButton>
                       }
                       targetOrigin={{ horizontal: 'right', vertical: 'top' }}
                       anchorOrigin={{ horizontal: 'right', vertical: 'top' }}
                       onTouchTap={this.showOptions}
                   >
                   </IconMenu>
                   <Popover
                       {...props}
                       animated={false}
                       style={{ float: 'right', position: 'relative', marginTop: '46px', marginLeft: leftGap }}
                       open={this.state.showOptions}
                       anchorEl={this.state.anchorEl}
                       anchorOrigin={{ horizontal: 'right', vertical: 'bottom' }}
                       targetOrigin={{ horizontal: 'right', vertical: 'top' }}
                       onRequestClose={this.closeOptions}
                   >
                       <TopRightMenuItems />
                       {cookies.get('loggedIn') ?
                           (<MenuItem primaryText='Botbuilder'
                               containerElement={<Link to='/botbuilder' />}
                               rightIcon={<Extension />} />) :
                           null
                       }
                       <MenuItem primaryText='Settings'
                           containerElement={<Link to='/settings' />}
                           rightIcon={<Settings />} />
                       {cookies.get('loggedIn') ?
                           (<MenuItem primaryText='Logout'
                               containerElement={<Link to='/logout' />}
                               rightIcon={<Exit />} />) :
                           (<MenuItem primaryText='Login'
                               onTouchTap={this.handleLogin}
                               rightIcon={<LoginIcon />} />)
                       }
                   </Popover>
               </div>
           </div>
       );
Handling the conditional display of menu items based on the user session

Some features are to be offered to only those who are logged in and thus we need to display them depending on the user session i.e they should be visible when the user is logged in and hidden when the user is logged out. User state is stored in the browser’s cookies and using that we can achieve the desired result.

{
     cookies.get('loggedIn') ?
       (<MenuItem primaryText="Botbuilder"
         href="https://skills.susi.ai/botbuilder"
         rightIcon={<Extension/>}/>):
         null
}

 

So I hope after going through this blog you have a much more clearer insight to how the StaticAppBar is implemented.

 

References

  1. Check out AppBar component from material-ui library here.
  2. Check blog post introducing the usage of material-ui in react here
Continue Reading

“STOP” action in SUSI Android App

Generally whenever there was a long query asked from SUSI through speech, it would respond to the user with a speech output, similarly the output is given through speech whenever the user clicks on the “Try it” button on the skill details activity.

The long answers for e.g. asking SUSI “How to cook biryani ?” gives a very long response which when conveyed through the speech output takes a long amount of time. Also most of the time users don’t want to listen to such long answers, but at the same time there is no option or feature to stop SUSI. The user either needs to switch over to another activity or has to close the app.

So, to solve this problem such that neither the user has to shut down the app nor the user has to switch over to another activity, the “STOP” action was added in SUSI.

The “STOP” action was integrated in the server and is of following type :

 

“actions”: [{“type”: “stop”}],

How to define STOP response ?

To integrate this action type in the app, a separate response type was added and checked for. In the file ParseSusiResponseHandler.kt the action type stop was added as :

Constant.STOP -> try {
  stop = susiResponse.answers[0].actions[1].type
} catch (e: Exception) {

}

So, now whenever the query of type “stop” was entered by the user it would be caught by this block and further processing could be done. The stop variable takes the value from the JSON response fetched and the value extracted is the one stored against the type key in the second field of the actions JSON in the first field of the answers JSON.

What to be done when executing STOP?

After the STOP action was caught the task to do was to define the behaviour of the app when this response was caught. So, first thing that should happen when STOP is received is that the TextToSpeech Engine should close so that SUSI no longer can speak the sentence but defining top is more than just closing the TTS engine. STOP signifies that any activity that is being continued right now should be stopped.

So, Using the android lifecycle methods I added a function in the IChatView interface to define what actions should take place in the stopping process.

override fun stopMic() {
  onPause()
  registerReceiver(networkStateReceiver, IntentFilter(ConnectivityManager.CONNECTIVITY_ACTION))

  window.setSoftInputMode(
          WindowManager.LayoutParams.SOFT_INPUT_STATE_ALWAYS_HIDDEN
  )

  if (recordingThread != null)
      chatPresenter.startHotwordDetection()

  if (etMessage.text.toString().isNotEmpty()) {
      btnSpeak.setImageResource(R.drawable.ic_send_fab)
      etMessage.setText(“”)
      chatPresenter.micCheck(false)
  }

  chatPresenter.checkPreferences()
}

The above function sends the activity to the onPause() lifecycle method to put the activity in a pausing state so that all that is taking place right now in the activity stops and this definitely serves our purpose. But after pausing the activity, there was a further need to perform some functionality that had to done on the start of the activity and therefore the code for that was also added in this function.

How to catch STOP?

The below code was added in ChatPresenter.kt file in which if the actionType from the psh object of type ParseSusiResponseHelper is “STOP” then the view function stopMic() is called which was defined above.

val psh = ParseSusiResponseHelper()
psh.parseSusiResponse(susiResponse, i, utilModel.getString(R.string.error_occurred_try_again))

var setMessage = psh.answer
if (psh.actionType == Constant.ANSWER && (PrefManager.checkSpeechOutputPref() && check || PrefManager.checkSpeechAlwaysPref())) {
  setMessage = psh.answer

  var speechReply = setMessage
  if (psh.isHavingLink) {
      speechReply = setMessage.substring(0, setMessage.indexOf(“http”))
  }
  chatView?.voiceReply(speechReply, susiResponse.answers[0].actions[i].language)
} else if (psh.actionType == Constant.STOP) {
  setMessage = psh.stop
  chatView?.stopMic()
}

Final Output

References

  1. Stop  json response from susi server : https://api.susi.ai/susi/chat.json?timezoneOffset=-330&q=susi+stop
  2. Android life cycle methods – Google: https://developer.android.com/guide/components/activities/activity-lifecycle
  3. Interaction between view and presenters : https://medium.com/@cervonefrancesco/model-view-presenter-android-guidelines-94970b430ddf
Continue Reading
Close Menu