Using SUSI.AI Accounting Model to store Device information on Server

Just like with Google Home devices and Amazon Alexa devices, SUSI.AI Users should have a way to add and store the information of their devices (Smart Speakers) on SUSI Server, so that it could be displayed to them on various clients. Hence, we needed a Servlet which could add and store the User data on the Server.

Implementation of Servlets in SUSI.AI

All servlets in SUSI extend AbstractAPIHandler class and implement APIHandler. All servlets have 4 methods, which we overwrite depending on what we want the servlet to do. They are as follows :

@Override
    public String getAPIPath() {
        return null;
    }

@Override
    public BaseUserRole getMinimalBaseUserRole() {
        return null;
    }

@Override
    public JSONObject getDefaultPermissions(BaseUserRole baseUserRole) {
        return null;
    }

@Override
    public ServiceResponse serviceImpl(Query post, HttpServletResponse response, Authorization rights, JsonObjectWithDefault permissions) throws APIException {
        Return null;
    }

 

How these 4 methods work together?

  1. First method is getAPIPath(). It returns the endpoint of the servlet.
  2. The second method is getMinimalBaseUserRole(). It returns the minimum privilege level required to access the endpoint.
  3. The third method is getDefaultPermissions(). It gets the Default Permissions of a UserRole in SUSI Server. Different UserRoles have different permissions defined in SUSI Server.
  4. Whenever the endpoint defined in the getAPIPath() method is called properly, it responds with whatever is defined in the fourth method, which is serviceImpl().

How is User data stored on SUSI Server?

Before we move on to the actual implementation of the API required to store the device information, we should get a brief idea of how exactly User data is stored on SUSI Server.

Every SUSI User has an Accounting Object. It is a JSONObject which stores the Settings of the particular User on the Server. For example, every time you change some setting on the Web Client https://chat.susi.ai, an API call is made to aaa/changeUserSettings.json with appropriate parameters with information of the changed setting, and the User settings are stored to the Server in the Accounting Object of the User.

Implementation of a Servlet to store Device information on Server

The task of this servlet is to store the information of a new device (Smart speaker) whenever it is initially set up using the Android/iOS app.

This is the implementation of the 4 methods of a servlet which is used to store information of connected devices:

    @Override
    public String getAPIPath() {
        return "/aaa/addNewDevice.json";
    }

    @Override
    public UserRole getMinimalUserRole() {
        return UserRole.USER;
    }

    @Override
    public JSONObject getDefaultPermissions(UserRole baseUserRole) {
        return null;
    }

    @Override
   public ServiceResponse serviceImpl(Query query, HttpServletResponse response, Authorization authorization, JsonObjectWithDefault permissions) throws APIException {

           JSONObject value = new JSONObject();
           
           String key = query.get("macid", null);
           String name = query.get("name", null);
           String device = query.get("device", null);
               
           if (key == null || name == null || device == null) {
               throw new APIException(400, "Bad service call, missing arguments");
           } else {
               value.put(name, device);
           }
           
           if (authorization.getIdentity() == null) {
               throw new APIException(400, "Specified user data not found, ensure you are logged in");
           } 
           else {
                Accounting accounting = DAO.getAccounting(authorization.getIdentity());
                if (accounting.getJSON().has("devices")) {
                       accounting.getJSON().getJSONObject("devices").put(key, value);
                   } 
                else {
                       JSONObject jsonObject = new JSONObject();
                       jsonObject.put(key, value);
                       accounting.getJSON().put("devices", jsonObject);
                   }
               
               JSONObject result = new JSONObject(true);
               result.put("accepted", true);
               result.put("message", "You have successfully added the device!");
               return new ServiceResponse(result);
           }
    }

 

As it can be seen from the above code, the endpoint for this servlet is /aaa/addNewDevice.json and it accepts 3 parameters –

  • macid : Mac Address of the device
  • name : Name of the device
  • device : Additional information which you want to send about the device (subject to changes)

As the main task of this servlet is user specific, and should only be accessible to the particular user, hence we returned UserRole as USER in the getMinimalUserRole() method.

In the serviceImpl() method, first we extract the value of the URL query parameters and store them in variables. If any of the parameters is missing, we display an error response code of 400 with error message “Bad service call, missing arguments”. If query parameters are fine, we store the name and device values in a new JSONObject, value in this case.

An if-else statement then checks for whether the User is logged in or not, using the authorization.getIdentity(), which is a function which returns the identity of the User. The implementation of this function is in Accounting.java file, and is as follows :

    public ClientIdentity getIdentity() {
        return identity;
    }

 

If the User is logged in, getAccounting() function of DAO.java file is called which returns an Accounting object according to the following implementation of the function :

    public static Accounting getAccounting(@Nonnull ClientIdentity identity) {
        return new Accounting(identity, accounting);
    }

 

Our device information is then stored in this Accounting object in the following format :

{
  "lastLoginIP": "162.158.166.19",
  "accepted": true,
  "message": "Success: Showing user data",
  "session": {"identity": {
    "type": "email",
    "name": "myemail@gmail.com",
    "anonymous": false
  }},
  "settings": {
    "customThemeValue": "4285f4,f5f4f6,fff,f5f4f6,fff,4285f4,",
    "theme": "light"
  },
  "devices": {
    "MacID2": {"MyDevice": "SmartSpeaker"},
    "MacID1": {"Name of device": "SmartSpeaker"}
  }
}

 

Resources

Continue ReadingUsing SUSI.AI Accounting Model to store Device information on Server

Fetching Skills using the ID of a user

The Dashboard on the SUSI Skill CMS is the page where all user specific Skill information is being displayed. It also has a section called “My Skills” which displays the Skills created by the user himself. The data for the “My Skills” section has to be fetched from the Skill metadata of each Skill. Hence, this blog post explains how the Skills created by an author are fetched using an API which takes author’s email as the only parameter.

Why is such an API required?

The task of displaying the “My Skills” section cannot be done entirely on the Client side with the use of existing APIs, as that would result in a lot of AJAX calls to the Server, because for each Skill, there would have to be an AJAX call for each user. Hence, we had to create an endpoint which could directly return the list of Skills created by a user using his ID. This would prevent multiple AJAX calls to the Server.

The endpoint used is:

"https://api.susi.ai/cms/getSkillsByAuthor.json?author_email="+author_email

 

Here, author_email is the email of the author who published the particular Skill, which is fetched from the Skill metadata object which looks like this:

"Antonyms": {
      // skill metadata
      "author_email": akjn11@gmail.com,
      // skill metadata
    }

Working of the API

The code to fetch the Skills created by an author is described in the GetSkillsByAuthor.java file. This API accepts either the author’s name, or the author’s email as the parameter. In our

We get all folders within a folder using the following method:

private void listFoldersForFolder(final File folder, ArrayList<String> fileList) {
        File[] filesInFolder = folder.listFiles();
        if (filesInFolder != null) {
            Arrays.stream(filesInFolder).filter(fileEntry -> fileEntry.isDirectory() && !fileEntry.getName().startsWith(".")).forEach(fileEntry->fileList.add(fileEntry.getName() + ""));
        }
    }

 

We use this method to get list of folders of all the group names for all the different Skills, and store them in a variable ‘folderList’ as per the following code:

File allGroup = new File(String.valueOf(model));
ArrayList<String> folderList = new ArrayList<String>();
listFoldersForFolder(allGroup, folderList);

 

For every folder corresponding to a group name, we traverse inside it and then further traverse inside the language folders to reach the place where all the Skill files are stored.

This is the code for the method to get all files within a folder:

private void listFilesForFolder(final File folder, ArrayList<String> fileList) {
    File[] filesInFolder = folder.listFiles();
    if (filesInFolder != null) {
        Arrays.stream(filesInFolder).filter(fileEntry -> !fileEntry.isDirectory() && !fileEntry.getName().startsWith(".")).forEach(fileEntry->fileList.add(fileEntry.getName() + ""));
    }
}

 

We need to get all Skill files inside the ‘languages’ folder, which itself is in ‘group’ folder. This is the code required to implement the same:

for (String temp_group_name : folderList){
        File group = new File(model, temp_group_name);
        File language = new File(group, language_name);
        ArrayList<String> fileList = new ArrayList<String>();
        listFilesForFolder(language, fileList);

        for (String skill_name : fileList) {
               // get skill metadata of each skill iteratively and put it in result JSONObject accordingly
            }
         }
    }

 

This is how the Skill metadata is fetched from the getSkillMetadata() method of SusiSkill class:

skill_name = skill_name.replace(".txt", "");
            JSONObject skillMetadata = SusiSkill.getSkillMetadata(model_name, temp_group_name, language_name, skill_name);

            if(skillMetadata.get("author_email").equals(author_email)) {
                skillMetadata.put("skill_name", skill_name);
                authorSkills.put(skillMetadata);

 

getSkillMetadata() method takes model of the Skill, group of the Skill, language of the Skill, Skill name and also duration (for usage statistical purposes) as the method parameters, and returns the metadata for the Skill in the format as shown below:

   "Antonyms": {
      "model": "general",
      "group": "Knowledge",
      "language": "en",
      "developer_privacy_policy": null,
      "descriptions": "A skill that returns the antonyms for a word",
      "image": "images/antonyms.png",
      "author": "akshat jain",
      "author_url": "https://github.com/Akshat-Jain",
      "author_email": akjn11@gmail.com,
      "skill_name": "Antonyms",
      "protected": false,
      "terms_of_use": null,
      "dynamic_content": true,
      "examples": ["Antonym of happy"],
      "skill_rating": {
        "negative": "0",
        "bookmark_count": 0,
        "positive": "0",
        "stars": {
          "one_star": 0,
          "four_star": 0,
          "five_star": 0,
          "total_star": 0,
          "three_star": 0,
          "avg_star": 0,
          "two_star": 0
        },
        "feedback_count": 0
      },
      "usage_count": 0,
      "skill_tag": "Antonyms",
      "creationTime": "2018-05-06T03:05:22Z",
      "lastAccessTime": "2018-06-23T18:48:36Z",
      "lastModifiedTime": "2018-05-06T03:05:22Z"
    }

 

This is how the Skills created by an author are fetched using this API. The JSON response of this endpoint shows the metadata of all Skills which have ‘author_email’ value the same as that given in the endpoint’s query parameter.

Resources

Continue ReadingFetching Skills using the ID of a user

Implementing API to Fetch all Ratings by a User on Different Skills on SUSI.AI Web Client

SUSI Skill CMS allows the users to rate any Skill on a scale of 1 to 5. The user can also provide a feedback to any Skill. This paves the path to implementing a Dashboard, which has all the analytic data of the user. Hence, an API needed to be implemented which could return the ratings done by a particular user on all different Skills.

How are servlets implemented in SUSI.AI?

All servlets in SUSI extend AbstractAPIHandler class and implement APIHandler. All servlets have 4 methods, which we overwrite depending on what we want the servlet to do. They are as follows :

    @Override
    public String getAPIPath() {
        return null;
    }

    @Override
    public BaseUserRole getMinimalBaseUserRole() {
        return null;
    }

    @Override
    public JSONObject getDefaultPermissions(BaseUserRole baseUserRole) {
        return null;
    }

    @Override
    public ServiceResponse serviceImpl(Query post, HttpServletResponse response, Authorization rights, JsonObjectWithDefault permissions) throws APIException {
        return null;
    }

 

How these 4 methods work together?

  • First method is getAPIPath(). It returns the endpoint of the servlet.
  • The second method is getMinimalBaseUserRole(). It returns the minimum privilege level required to access the endpoint.
  • The third method is getDefaultPermissions(). It gets the Default Permissions of a UserRole in SUSI Server. Different UserRoles have different permissions defined in SUSI Server.
  • Whenever the endpoint defined in the getAPIPath() method is called properly, it responds with whatever is defined in the fourth method, which is serviceImpl().

Implementing a servlet to fetch all ratings by a user on different Skills

The task of this servlet is to fetch all the ratings done by a user on all the different Skills, so that this fetched data could be used later on for implementation of various user specific features like Dashboard page. This is the implementation of the 4 methods of this servlet:

    @Override
    public UserRole getMinimalUserRole() {
        return UserRole.USER;
    }

    @Override
    public JSONObject getDefaultPermissions(UserRole baseUserRole) {
        return null;
    }

    @Override
    public String getAPIPath() {
        return "/cms/getProfileDetails.json";
    }

    @Override
    public ServiceResp onse serviceImpl(Query query, HttpServletResponse response, Authorization authorization, final JsonObjectWithDefault permissions) throws APIException {

    JsonTray fiveStarSkillRating = DAO.fiveStarSkillRating;
    // JSONObject and JsonArray Declarations

    // Checking if access_token has been given and is valid
    if (authorization.getIdentity() == null) {
        throw new APIException(400, "Specified user data not found, ensure you are logged in");
    } 


    // Fetching email of the user from access token and storing it in a string
    String email = authorization.getIdentity().getName();


    // Iterating over the fiveStarSkillRating JsonTray by extracting keys at every level and accessing their child objects through the extracted keys
    for(String model_name : fiveStarSkillRating.keys())
    {
        // Storing the list of group names and iterating over them
        for(String group_name : groupnameKeysList)
        {
            // Storing the list of language names and iterating over them
            for(String language_name : languagenameKeysList)
            {
                // Storing the list of skill names and iterating over them
                for(String skill_name : skillnamesKeysList)
                {
                    skillnameArray = languageObject.getJSONArray(skill_name);
                    // Iterating over the different Skill JSONObjects
                    for(int i=0; i<skillnameArray.length(); i++) {
                        String jsonEmail = skillnameArray.getJSONObject(i).get("email").toString();
                        if(jsonEmail.equals(email)) {
                        // If the Skill has been rated by the required author, then put the Skill JSONObject into the result array
                        }
                    }
                }
            }
        }
    }

    if(result.length()==0) {
        // Handling case when user hasn’t rated any Skill yet
        result.put("accepted", false);
        result.put("message", "User has not rated any Skills yet.");
        return new ServiceResponse(result);
    }
    result.put("accepted", true);
    result.put("message", "User ratings fetched.");
    return new ServiceResponse(result);
    }

 

As it can be seen from the above code, the endpoint for this servlet is /cms/getProfileDetails.json and it requires 1 parameter – the access token of the user.

As the main task of this servlet is user specific, and should only be accessible to the particular user, hence we returned UserRole as USER in the getMinimalUserRole() method.

In the serviceImpl() method, we iterate over the fiveStarSkillRating.json JsonTray and keep on extracting keys and accessing the corresponding JSONObjects until we reach the lowermost layer, where all the Skill names are listed. Iterating over the JSONObjects corresponding to each Skill, we check the email in the Skill JSONObject to identify the user. If the email present in the Skill JSONObject matches with the email corresponding to the access token provided as a query parameter, we extract rating and the timestamp for the Skill and put it in a JSONArray. Then, finally we put the JSONArray into our resulting JSONObject.

This is how we can fetch all the ratings by a user on all the different Skills.

Resources

 

Continue ReadingImplementing API to Fetch all Ratings by a User on Different Skills on SUSI.AI Web Client

Allowing user to submit ratings for skills in SUSI iOS

Rating is a great way to get feedback from the user. Generally, the 5-Star rating system used to get feedback. The Five-Star Quality Rating System was developed as an easy-to-understand rating system for users.

In SUSI iOS we are using the Star Rating field to get feedback of SUSI skills. We enable the user to rate the skills from one to five star. In the rating submission, we get the number of stars user picked.

The stars help show if you would recommend the skill to others. The values start at 1 (being the lowest) and go to 5 (being the highest), as seen below –

Server-Side Implementation –

fiveStarRatings API is using to submit users rating. Whenever the user taps the star fiveStarRatings being called:

func submitRating(_ params: [String: AnyObject], _ completion: @escaping(_ updatedRatings: Ratings?, _ success: Bool, _ error: String?) -> Void) {
let url = getApiUrl(UserDefaults.standard.object(forKey: ControllerConstants.UserDefaultsKeys.ipAddress) as! String, Methods.fiveStarRateSkill)
_ = makeRequest(url, .get, [:], parameters: params, completion: { (results, message) in
// handle request
....
})
}

The following params we send in the request:

  • Model
  • Group
  • Language
  • Skill
  • Stars
  • Access Token

After successfully rating submission we get updated ratings for each star of the particular skill. The following response we get from the server after successfully submitted rating:

{
"ratings": {
"one_star": 0,
"four_star": 1,
"five_star": 0,
"total_star": 1,
"three_star": 0,
"avg_star": 4,
"two_star": 0
},
"session": {"identity": {
"type": "email",
"name": "abc@a.com",
"anonymous": false
}},
"accepted": true,
"message": "Skill ratings updated"
}

We use ratings object from fiveStarRatings API to update the ratings displayed on charts and labels and also, we use ratings object to update Skill model which we passed from SkillListingController to SkillDetailController so the user can see updated rating chart when clicking back to skill.

func updateSkill(with ratings: Ratings) {..}

If the user already submitted ratings for a skill, we are using getRatingByUser API with same params as in fiveStarRatings except ratings to get already submitted user rating and we display that ratings as initial ratings in UI.

Implementation of Submit Rating UI –

RatingView is behind the submit rating UI. FloatRatingViewDelegate protocol is implemented to get ratings is updating or get updated.

@objc public protocol FloatRatingViewDelegate {
/// Returns the rating value when touch events end
@objc optional func floatRatingView(_ ratingView: RatingView, didUpdate rating: Double)

/// Returns the rating value as the user pans
@objc optional func floatRatingView(_ ratingView: RatingView, isUpdating rating: Double)
}

Rating updated on rating display chart:

Now see, how we handle the case when Skill is not rated before and the user first time rate the skill.

There is a label that shows the “Skill not rated yet” when a skill is not rated. When the user rates the skill, the label is hidden and chart bar and the label is shown.

if self.ratingsBackViewHeightConstraint.constant == 0 {
self.ratingsBackViewHeightConstraint.constant = 128.0
self.contentType.topAnchor.constraint(equalTo: self.ratingBackView.bottomAnchor, constant: 16).isActive = true
self.ratingsBackStackView.isHidden = false
self.topAvgRatingStackView.isHidden = false
self.notRatedLabel.isHidden = true
}

 

Resources –

  1. Swift Protocol: https://docs.swift.org/swift-book/LanguageGuide/Protocols.html
  2. SUSI Skills: https://skills.susi.ai/
  3. SUSI Server Link: https://github.com/fossasia/susi_server
  4. SUSI iOS Link: https://github.com/fossasia/susi_iOS
Continue ReadingAllowing user to submit ratings for skills in SUSI iOS

Implementing API to facilitate changing review status of a Skill

As any registered user can make Skills in SUSI Skill CMS, sometimes the Skill made is not up to the mark to be displayed on the CMS site. There needs to be a feature implemented where the Admin and higher user roles can review a Skill and approve it accordingly. Then only the approved Skills should be displayed on the Skill CMS. This feature required implementation of an API to allow Admin and higher user roles to change the review status of a Skill. This blog post explains how such an API has been implemented.

Implementing a servlet to allow changing review status of a Skill

The basic task of the servlet is to allow Admin and higher user roles to allow changing the review status of a Skill. The review status of a Skill can either be ‘true’, indicating that the Skill has been approved by the Admin, or ‘false’, which would indicate that the Skill still needs some improvements before it is actually displayed on the SUSI Skill CMS.

Here is the implementation of the API:

  1. The API should be usable to only the users who have a user role Admin or higher. Only those with minimum Admin rights should be allowed to control what Skills are displayed on the CMS site. This is implemented as follows:
   @Override
    public UserRole getMinimalUserRole() {
        return UserRole.ADMIN;
    }

 

  1. The endpoint for the API is ‘/cms/changeSkillStatus.json’. This is implemented as follows:
   @Override
    public String getAPIPath() {
        return "/cms/changeSkillStatus.json";
    }

 

  1. The main method of the servlet is the serviceImpl() method. This is where the actual code goes which will be executed each time the API is called. This is implemented as follows:

    @Override
    public ServiceResponse serviceImpl(Query call, HttpServletResponse response, Authorization authorization, JsonObjectWithDefault permissions) throws APIException {

       // Store the query parameters in variables

        if(skill_name == null || !(reviewed.equals("true") || reviewed.equals("false"))) {
            throw new APIException(400, "Bad service call, missing arguments.");
        }

        JSONObject result = new JSONObject();
        JsonTray skillStatus = DAO.skillStatus;
        JSONObject modelName = new JSONObject();
        JSONObject groupName = new JSONObject();
        JSONObject languageName = new JSONObject();
        JSONObject skillName = new JSONObject();

        if(reviewed.equals("true")) {
            JSONObject reviewStatus = new JSONObject();
            reviewStatus.put("reviewed", true);
                // traverse through the skillStatus.json file down to the Skill name and put review status ‘true’ in it
                skillName.put("reviewed", true);
                result.put("accepted", true);
                result.put("message", "Skill review status changed successfully.");
                return new ServiceResponse(result);
            }


            // deleting skill name from skillStatus.json file if it’s review status is being changed to ‘false’
            else {
                if (skillStatus.has(model_name)) {
                    modelName = skillStatus.getJSONObject(model_name);
                    if (modelName.has(group_name)) {
                        groupName = modelName.getJSONObject(group_name);
                        if (groupName.has(language_name)) {
                          languageName = groupName.getJSONObject(language_name);
                            if (languageName.has(skill_name)) {
                                languageName.remove(skill_name);
                                    skillStatus.remove(model_name);
                                }
                                skillStatus.commit();
                            }
                        }
                    }
                }
                result.put("accepted", true);
                result.put("message", "Skill review status changed successfully.");
                return new ServiceResponse(result);
            }
        }

 

The list of reviewed Skills is being stored in the ‘skillStatus.json’ file. So we need to traverse through that file and store the review status of the Skill as required while making the API call.

The API takes 5 parameters:

  • Model of the Skill
  • Group of the Skill
  • Language of the Skill
  • Skill name
  • Review status to be set for the Skill – true or false

On making the API call, if the value of the query parameter ‘reviewed’ is ‘true’, then the Skill name along with its review status is being appended in the ‘skillStatus.json’ file. This is done by traversing through the file using the specified model, group and language info of the Skill.

However, if the value of the query parameter ‘reviewed’ is ‘false’, then we need to check if the Skill is already there in the ‘skillStatus.json’ file. If it is already there, then we remove its entry from the file. If it isn’t in the file already, then it’s review status is already ‘false’. We also need to commit this change to the JsonTray for it to get reflected in the Server.

This is how an API has been implemented which would allow Admin and higher user roles to change the review status of any Skill, which would then facilitate showing only the approved Skills on the CMS site.

  1. Resources

     

Continue ReadingImplementing API to facilitate changing review status of a Skill

Adding a Horizontally scrollable component to display Skills based on metrics

In this blog post, I will discuss about the implementation of a horizontally scrollable component to display skill based on metrics. The purpose of the implementation is to show top skills based on metrics related to usage, ratings, etc in SUSI.AI Skills CMS.

Implementational details

  • We call this component SkillCardScrollList which takes in a list of cards to be displayed along with some other properties and returns an UI, as shown in the above GIF.
  • The parameters that the component takes are:
    • scrollId: It is a required field of the type String. It is the id name of the horizontally scrollable div.
    • skills: It contains an array of cards that are to displayed inside the container.
    • languageValue: It represents the language of the skills that are shown.
    • skillUrl: It contains the URL that the app would be taken to, on clicking individual Skill Card.
    • modelValue: It contains the model that the skill belongs to.
  • Here is a sample of how it is used in the BrowseSkill component, for showing the Top Rated Skills in a SkillCardsScrollList
<SkillCardScrollList
    scrollId="topRated"
    skills={this.state.topRatedSkills}
    modalValue={this.state.modalValue}
    languageValue={this.state.languageValue}
    skillUrl={this.state.skillUrl}
/>

 

  • The reason behind passing an unique scrollId as a prop to the component is that, there was a need to trigger the scroll event of the scrollable div n the click of left and right Floating Action Buttons (FABs) as shown in the UI. And, on multiple imports of this component, there would have been inconsistent scroll behaviour seen, had it not been unique.
  • Following in the code block of the component, which will be explained in details, that deals with the main implementation –
.
.
.
.
  scrollLeft = () => {
    let parentEle = document.getElementById(this.props.scrollId);
    let scrollValue = $(parentEle).scrollLeft() - 200;
    $(parentEle)
      .stop()
      .animate({ scrollLeft: scrollValue }, 100);
  };

  scrollRight = () => {
    // Similar function of scrollLeft
  };

  loadSkillCards = () => {
    let cards = [];
    Object.keys(this.state.skills).forEach(el => {
      .
      /* Each skill object is passed and then pushed to the cards
        array*/
      .
      );
    });
    // Set the cards array in the state 
    this.setState({
      cards,
    });
  };

  render() {
    return (
      <div
        style={{
          marginTop: '20px',
          marginBottom: '40px',
          textAlign: 'justify',
          fontSize: '0.1px',
          width: '100%',
        }}
      >
        <div>
          <div
            id={this.props.scrollId}
            className="scrolling-wrapper"
            style={styles.gridList}
          >
            <FloatingActionButton
              mini={true}
              backgroundColor={'#4285f4'}
              style={styles.leftFab}
              onClick={this.scrollLeft}
            >
              <NavigationChevronLeft />
            </FloatingActionButton>
            {this.state.cards}
            <FloatingActionButton
              mini={true}
              backgroundColor={'#4285f4'}
              style={styles.rightFab}
              onClick={this.scrollRight}
            >
              <NavigationChevronRight />
            </FloatingActionButton>
          </div>
        </div>
      </div>
    );
  }
}

 

  • The div with class scrolling-wrapper is actually scrolled on the click of the left and right FAB. For choosing the correct div to be scrolled, there was a necessary condition of an unique id as explained earlier, which has been set to the div.
  • For making the component horizontally scrollable, specific CSS rules are added to the div. They are –
gridList: {
  margin: '10px',
  textAlign: 'center',
  overflowX: 'scroll',
  overflowY: 'hidden',
  whiteSpace: 'nowrap',
},
leftFab: {
  position: 'absolute',
  left: 260,
  marginTop: 75,
},
rightFab: {
  position: 'absolute',
  right: 0,
  marginTop: 75,
  marginRight: 10,
},

 

  • The CSS rules for the FABs make them fixed in a position and only lets the card list scroll.
  • Lastly, there was a issue regarding the presence of horizontal scroll-bar been shown, which makes the UI look a bit unpleasant.

  • It was hidden with a pseudo selector CSS rule.
div.scrolling-wrapper::-webkit-scrollbar {
    display: none;
}

 

This was the implementation for the horizontally scrollable component for displaying Skill List based on a standard metrics. I hope, you found the blog helpful in making the understanding of the implementation better.

Resources

Continue ReadingAdding a Horizontally scrollable component to display Skills based on metrics

Change Text-to-Speech Voice Language of SUSI in SUSI iOS

SUSI iOS app now enables the user to change the text-to-speech voice language within the app. Now, the user can select any language of their choice from the list of 37 languages list. To change the text-to-speech voice language, go to, Settings > Change SUSI’s Voice, choose the language of your choice. Let see here how this feature implemented.

Apple’s AVFoundation API is used to implement the text-to-speech feature in SUSI iOS. AVFoundation API offers 37 voice languages which can be used for text-to-speech voice accent. AVFoundation’s AVSpeechSynthesisVoice API can be used to select a voice appropriate to the language of the text to be spoken or to select a voice exhibiting a particular local variant of that language (such as Australian or South African English).

To print the list of all languages offered by AVFoundation:

import AVFoundation

print(AVSpeechSynthesisVoice.speechVoices())

Or the complete list of supported languages can be found at Languages Supported by VoiceOver.

When the user clicks Change SUSI’s voice in settings, a screen is presented with the list of available languages with the language code.

Dictionary holds the list of available languages with language name and language code and used as Data Source for tableView.

var voiceLanguagesList: [Dictionary<String, String>] = []

When user choose the language and click on done, we store language chosen by user in UserDefaults:

UserDefaults.standard.set(voiceLanguagesList[selectedVoiceLanguage][ControllerConstants.ChooseLanguage.languageCode], forKey: ControllerConstants.UserDefaultsKeys.languageCode)
UserDefaults.standard.set(voiceLanguagesList[selectedVoiceLanguage][ControllerConstants.ChooseLanguage.languageName], forKey: ControllerConstants.UserDefaultsKeys.languageName)

Language name with language code chosen by user displayed in settings so the user can know which language is currently being used for text-to-speech voice.

To select a voice for use in speech, we obtain an AVSpeechSynthesisVoice instance using one of the methods in Finding Voices and then set it as the value of the voice property on an AVSpeechUtterance instance containing text to be spoken.

Earlier stored language code in UserDefaults shared instance used for setting the text-to-speech language for AVSpeechSynthesisVoice.

if let selectedLanguage = UserDefaults.standard.object(forKey: ControllerConstants.UserDefaultsKeys.languageCode) as? String {
speechUtterance.voice = AVSpeechSynthesisVoice(language: selectedLanguage)
}

AVSpeechUtterance is responsible for a chunk of text to be spoken, along with parameters that affect its speech.

Resources –

  1. UserDefaults: https://developer.apple.com/documentation/foundation/userdefaults
  2. AVSpeechSynthesisVoice: https://developer.apple.com/documentation/avfoundation/avspeechsynthesisvoice
  3. AVFoundation: https://developer.apple.com/av-foundation/
  4. SUSI iOS Link: https://github.com/fossasia/susi_iOS
Continue ReadingChange Text-to-Speech Voice Language of SUSI in SUSI iOS

STOP action implementation in SUSI iOS

You may have experienced, you can stop Google home or Amazon Alexa during the ongoing task. The same feature is available for SUSI too. Now, SUSI can respond to ‘stop’ action and stop ongoing tasks (e.g. SUSI is narrating a story and if the user says STOP, it stops narrating the story). ‘stop’ action is introduced to enable the user to make SUSI stop anything it’s doing.

Video demonstration of how stop action work on SUSI iOS App can be found here.

Stop action is implemented on SUSI iOS, Web chat, and Android. Here we will see how it is implemented in SUSI iOS.

When you ask SUSI to stop, you get following actions object from server side:

"actions": [{"type": "stop"}]

Full JSON response can be found here.

When SUSI respond with ‘stop’ action, we create a new action type ‘stop’ and assign `Message` object `actionType` to ‘stop’.

Adding ‘stop’ to action type:

enum ActionType: String {
... // other action types
case stop
}

Assigning to the message object:

if type == ActionType.stop.rawValue {
message.actionType = ActionType.stop.rawValue
message.message = ControllerConstants.stopMessage
message.answerData = AnswerAction(action: action)
}

A new collectionView cell is created to respond user with “stoped” text.

Registering the stopCell:

collectionView?.register(StopCell.self, forCellWithReuseIdentifier: ControllerConstants.stopCell)

Add cell to the chat screen:

if message.actionType == ActionType.stop.rawValue {
if let cell = collectionView.dequeueReusableCell(withReuseIdentifier: ControllerConstants.stopCell, for: indexPath) as? StopCell {
cell.message = message
let message = ControllerConstants.stopMessage
let estimatedFrame = self.estimatedFrame(message: message)
cell.setupCell(estimatedFrame, view.frame)
return cell
}
}

AVFoundation’s AVSpeechSynthesizer API is used to stop the action:

func stopSpeakAction() {
speechSynthesizer.stopSpeaking(at: AVSpeechBoundary.immediate)
}

This method immediately stops the speak action.

Final Output:

Resources – 

  1. About SUSI: https://chat.susi.ai/overview
  2. JSON response for ‘stop’ action: https://api.susi.ai/susi/chat.json?timezoneOffset=-330&q=susi+stop
  3. AVSpeechSynthesisVoice: https://developer.apple.com/documentation/avfoundation/avspeechsynthesisvoice
  4. AVFoundation: https://developer.apple.com/av-foundation/
  5. SUSI iOS Link: https://github.com/fossasia/susi_iOS
  6. SUSI Android Link: https://github.com/fossasia/susi_android
  7. SUSI Web Chat Link: https://chat.susi.ai/
Continue ReadingSTOP action implementation in SUSI iOS

Implementing the Feedback section on Skills CMS

In this blog post, we are going to implement the Skill feedback system on the Skills CMS. The features that are added by this implementation are displaying all the comments/feedbacks of the user, ability to add new feedback and also option to edit a previous feedback that was added.

The UI interacts with the back-end server via two APIs –

Detailed explanation of the implementation

  • The first task was to create a separate component for the feedback section – SkillFeedbackCard.js, along with the CSS file SkillFeedbackCard.css
  • Create ES6 function to get all the Feedbacks of a skill,namely getFeedback(), on the parent component, i.e, SkillListing.js
getFeedback = () => {
    let getFeedbackUrl = `${urls.API_URL}/cms/getSkillFeedback.json`;
    let modelValue = 'general';
    this.groupValue = this.props.location.pathname.split('/')[1];
    this.languageValue = this.props.location.pathname.split('/')[3];
    getFeedbackUrl = getFeedbackUrl + '?model=' + modelValue + '&group=' + this.groupValue + '&language=' + this.languageValue + '&skill=' + this.name;

    let self = this;
    // Get skill feedback of the visited skill
    $.ajax({
        url: getFeedbackUrl,
        dataType: 'jsonp',
       crossDomain: true,
        jsonp: 'callback',
        success: function (data) {
            self.saveSkillFeedback(data.feedback);
        },
        error: function(e) {
            console.log(e);
        }
    });
};

saveSkillFeedback = (feedback = []) => {
    this.setState({
        skill_feedback: feedback
    })
}

 

  • This above code contains the function getFeedback(), that makes an API call to the server for getting all the feedbacks. On successfully getting the response, the feedback array of the response is then passed to a function, saveSkillFeedback(), which in turn updates the skill_feedback state, which was declared in the constructor. This re-renders the components and displays the feedback in the UI.
{
  "feedback": [
    {
      "feedback": "Awesome skill!",
      "email": "coolakshat24@gmail.com",
      "timestamp": "2018-06-12 19:28:39.297"
    },
    {
      "feedback": "Awesome skill!",
      "email": "akshatnitd@gmail.com",
      "timestamp": "2018-06-12 21:35:53.048"
    }
  ],
  "session": {"identity": {
    "type": "host",
    "name": "141.101.98.18_a7ab9c4d",
    "anonymous": true
  }},
  "skill_name": "aboutsusi",
  "accepted": true,
  "message": "Skill feedback fetched"
}

 

  • Then, we go ahead and create the function that is responsible for posting new feedback and editing them as well.
postFeedback = (newFeedback) => {

    let baseUrl = urls.API_URL + '/cms/feedbackSkill.json';
    let modelValue = 'general';
    this.groupValue = this.props.location.pathname.split('/')[1];
    this.languageValue = this.props.location.pathname.split('/')[3];
    let postFeedbackUrl = baseUrl + '?model=' + modelValue + '&group=' + this.groupValue + '&language=' + this.languageValue + '&skill=' + this.name + '&feedback=' + newFeedback + '&access_token='+cookies.get('loggedIn');

    let self = this;
    $.ajax({
        url: postFeedbackUrl,
        dataType: 'jsonp',
        jsonp: 'callback',
        crossDomain: true,
        success: function (data) {
            self.getFeedback()
        },
        error: function(e) {
            console.log(e);
        }
    });
};

 

  • This above code snippet contains the function postFeedback(newFeedback), that takes the user feedback and make an API call to update it on the server.
  • All the required functions are ready. Now we add the SkillFeedbackCard.js component on the SkillListing.js component and pass useful data in the props.
<SkillFeedbackCard
    skill_name={this.state.skill_name}
    skill_feedback={this.state.skill_feedback}
    postFeedback={this.postFeedback}
/>
  • The next step is creating the UI for the SkillFeedbackCard.js component. We have used standard Material-UI components for creating the UI, that includes List, ListItem, Divider, IconButton, etc.
  • Code snippet for the Feedback ListItem  –
<ListItem
    key={index}
    leftAvatar={<CircleImage name={data.email.toUpperCase()} size='40' />}
    primaryText={data.email}
    secondaryText={<p> {data.feedback} </p>}
/>

 

  • The next part of the UI implementation creating option to edit and post feedback.
  • Code snippet for the Post feedback section  –
className=“feedback-textbox”> id=“post-feedback” hintText=“Skill Feedback” defaultValue=“” errorText={this.state.errorText} multiLine={true} fullWidth={true} /> label=“Post” primary={true} backgroundColor={‘#4285f4’} style={{ margin: 10 }} onClick={this.postFeedback} />

 


 

  • For the edit section, I have used a Dialog box for it. Code snippet for the Edit feedback section  –
<Dialog
    title="Edit Feedback"
    actions={actions}
    modal={false}
    open={this.state.openDialog}
    onRequestClose={this.handleClose}
>
    <TextField
        id="edit-feedback"
        hintText="Skill Feedback"
        defaultValue={userFeedback}
        errorText={this.state.errorText}
        multiLine={true}
        fullWidth={true}
    />
</Dialog>

 

This was the implementation for the Skill Feedback System on the Skills CMS and I hope, you found the blog helpful in making the understanding of the implementation better.

Resources

 

Continue ReadingImplementing the Feedback section on Skills CMS

How does live preview work on SUSI AI chatbot wizard

Live preview is an important feature for the chatbot building process in SUSI.AI botbuilder. It tells the bot creator how the bot looks and behaves at any time.

We use iframe for creating a live preview.

What is iframe?

Frames allow browser window to be split into separate segments. All these segments can be used for displaying a different webpage within the browser window.

is an HTML tag. iframe stands for “inline frame”. It places another HTML document in a frame. The content of the element is used as an alternative text to be displayed if the browser does not support inline frames.

This was first introduced by Microsoft Internet Explorer in 1997, standardized in HTML 4.0 transitional and allowed in HTML5.

Syntax:

<iframe src=”URL”></iframe>

Some attributes:

  • Height
  • Width
  • Name
  • Id

Code used for preview of chatbot screen:

<iframe title="botPreview" name="frame-1" id="frame-1" src={locationBot} height="600px" style={styles.iframe}></iframe>

Code used for preview of chatbot avatar:

<iframe title="botAvatarPreview" name="frame-2" id="frame-2" src={locationAvatar} height="100px" style={styles.iframe}></iframe>

Preview bot fetches the current theme status and skill status from SUSI server and then applies it to the chatbot. Now, this details will be unique to the user logged in. Hence, we need to somehow identify the person logged in. This is done using access token stored in the cookies which is accessed through cookies.get(‘loggedIn’).

How to pass access token to iframe?

This is the major issue faced when creating preview using iframe because even though the HTML window in iframe is rendered on the same parent page, we can not access the elements of HTML page inside iframe. This is because iframe loads dynamically. There are few solutions for this problem.

One of them is to create a script in iframe that would run when iframe is loaded. Now this script can access the elements of parent window (the one containing iframe) using window.parent.document.getElementById(‘#target’)

The best solution is however to pass the access token in the URL of the source of iframe. Then this can be accessed by HTML page in frame using location.href .

Code sample to pass access token in url:

src = '/BotPreview.html?access='+cookies.get('loggedIn')

Now, we have to split this URL and get the access token. This can be easily done using the following script.

var url = location.href.split(‘?’)[1].split(‘=’);
var access_token = url[1];

Here in this example, the url variable firstly splits the URL at ‘?’. This creates an array with two elements.

[‘url.com/BotPreview.html’, ‘access=abcdef123’]

Then we access the element on index 1, i.e. ‘access=abcdef123’ and split it at the ‘=’. This further creates an array of two elements.

[‘access’, ‘abcdef123’]

Now, we can access the access token which is at index 1 in this array.

References

Continue ReadingHow does live preview work on SUSI AI chatbot wizard