SUSI AI 5 Star Skill Rating System

For making a system more reliable and robust, continuous evaluation is quite important. So is in case of SUSI AI. User feedback is important to improve SUSI skills and create new ones. Previously we had only thumbs up / thumbs down as a feedback method, from the susi chat client. But now a 5 star rating system has been added to the SUSI Skill CMS so that users can rate a skill there. Before the implementation of API  let’s look how data is stored in SUSI AI Susi_server uses DAO in which skill rating is stored as JSONTray.

The server side implementation

A new java class has been created for the API, FiveStarRateSkillService.java.

public class FiveStarRateSkillService extends AbstractAPIHandler implements APIHandler {
    private static final long serialVersionUID =7947060716231250102L;
    @Override
    public BaseUserRole getMinimalBaseUserRole() {
        return BaseUserRole.USER;
    }
    @Override
    public JSONObject getDefaultPermissions(BaseUserRole baseUserRole) {
        return null;
    }
    @Override
    public String getAPIPath() {
        return "/cms/rateSkill.json";
    }
...
}

The getMinimalBaseRole method tells the minimum User role required to access this servlet it can also be ADMIN, USER or ANONYMOUS. In our case it is USER. A user needs to be logged in to rate a skill on a scale of 1-5 stars.  The API runs at “/cms/fiveStarRateSkill.json” endpoint.

Next, create serviceImpl method in the above class to handle the request from the client and respond to it.

  1. Fetch the required query parameters and store them in variables. They include skill model, group, language, skill name and starts that the user has given in the rating.

String skill_name = call.get("skill", null);
String skill_stars = call.get("stars", null);
  1. Then check if the skill exists. If not them throw an exception. Otherwise, increment the count of the corresponding rating. The rating object has keys as one_star, two_star, three_star, four_star and five_star that has the count of that star rating.       

if (skill_stars.equals("1")) {
    skillName.put("one_star", skillName.getInt("one_star") + 1 + "");
}
else if (skill_stars.equals("2")) {
    skillName.put("two_star", skillName.getInt("two_star") + 1 + "");
}
else if (skill_stars.equals("3")) {
    skillName.put("three_star", skillName.getInt("three_star") + 1 + "");
}
else if (skill_stars.equals("4")) {
    skillName.put("four_star", skillName.getInt("four_star") + 1 + "");
}
else if (skill_stars.equals("5")) {
    skillName.put("five_star", skillName.getInt("five_star") + 1 + "");
}
  1. Re-calculate the total rating done on that skill and its average rating and update the object. If the skill has not been already rated then create a new rating object and initialize it with the 0 star counts.

public JSONObject createRatingObject(String skill_stars) {
        JSONObject skillName = new JSONObject();
        JSONObject skillStars = new JSONObject();

        skillStars.put("one_star", 0);
        skillStars.put("two_star", 0);
        skillStars.put("three_star", 0);
        skillStars.put("four_star", 0);
        skillStars.put("five_star", 0);
        skillStars.put("avg_star", 0);
        skillStars.put("total_star", 0);

        skillName.put("stars", skillStars);
}

The complete FiveStarRateSkillService.java is available here : –

https://github.com/fossasia/susi_server/blob/development/src/ai/susi/server/api/cms/FiveStarRateSkillService.java

Rating a skill

Sample endpoint

https://api.susi.ai/cms/fiveStarRateSkill.json?model=general&group=Knowledge&language=en&skill=aboutsusi&stars=3&callback=p&_=1526813916145

This gives 3 star rating to the “aboutsusi” skill.

Parameters

  • Model
  • Group
  • Language
  • Skill
  • Stars

Response

{
  "ratings": {
    "one_star": 0,
    "four_star": 0,
    "five_star": 1,
    "total_star": 1,
    "three_star": 0,
    "avg_star": 5,
    "two_star": 0
  },
  "session": {"identity": {
    "type": "email",
    "name": "[email protected]",
    "anonymous": false
  }},
  "accepted": true,
  "message": "Skill ratings updated"
}

Getting the stats of Skill Ratings

Sample endpoint

http://127.0.0.1:4000/cms/getSkillRating.json?model=general&group=Knowledge&language=en&skill=aboutsusi&callback=p&_=1526813916145

This fetches the current ratings of the “aboutsusi” skill.

Parameters

  • Model
  • Group
  • Language
  • Skill

Response

{
    "session": {
        "identity": {
            "type": "host",
            "name": "172.68.144.159_81c88a10",
            "anonymous": true
        }
    },
    "skill_name": "aboutsusi",
    "accepted": true,
    "message": "Skill ratings fetched",
    "skill_rating": {
        "negative": "0",
        "positive": "0",
        "stars": {
            "one_star": 0,
            "four_star": 2,
            "five_star": 1,
            "total_star": 4,
            "three_star": 1,
            "avg_star": 4,
            "two_star": 0
        },
        "feedback_count": 3
    }
}

Conclusion

So this 5 star rating system will help in improving the SUSI skills. Also, it will help in making better decisions when we have multiple similar skills and we have to choose one to respond to the user query.

References

How search works on Susi Skill CMS

The SUSI Skill CMS is a central dashboard where the developers can add skills to make SUSI more intelligent. One of the major thing that was missing there was a search bar. So I recently added one that can search a skill based on:

  • Skill name
  • Skill description
  • Author name
  • Examples of the skill

How to add the search bar?

  1. Install the Material Search Bar component from the terminal.

npm i --save material-ui-search-bar

 

  1. Import this component in the BrowseSkill.js file.

import SearchBar from 'material-ui-search-bar'

 

  1. Create a state variables for the search query and initialize it to an empty string.

this.state = {
	...
	searchQuery:''
	...
};

 

  1. Add the search bar (UI Part) below the filters on CMS. Add a listener function to it that is called when the value of the search query changes.

<SearchBar
     onChange={this.handleSearch}
     value={this.state.searchQuery}
   />

 

  1. The search handler : Create a handler (handleSearch) to that listens to onChange events on the SearchBar. This function sets the value of searhQuery state variable and loads the card again based on the search filter.

handleSearch
  = (value) => {
    this.setState({searchQuery: value}, function () {
    this.loadCards();
    });
};

 

  1. The loadCards() function : The loadCards() function send a request to the Susi Server which in turn send a json response. Then this function makes cards for every skill and adds them to the CMS. Modify the loadCards() function to filter the cards array based on the search query. The javascript string function match() is used to check if the skill name, description, author’s name or examples match the search query. The filter() function adds the skill card to the filtered data if the match() function returns true (i.e., the skill is relevant to the search query).

How the filter works?

  1. First check if there’s something to search for. If the searchQuery.length is equal to zero then that means that there is nothing to search for.

self.state.searchQuery.length >0

 

  1. Then filter the related results based on
  • Skill name : The filter() function adds the skill card to the filtered data if the match function returns true (i.e., the skill is relevant to the search query). The match() function retuns true if the skill name matches the search query.

if (i.skill_name) {
    	result =  i.skill_name.toLowerCase()
        	.match( self.state.searchQuery.toLowerCase() );
    	if (result) {
        	return result;
    	}
}

 

Similarly, filter the cards on the basis of skill description and skill author’s name.

  • Skill examples: Loop over all the skill examples and check if any example matches the search query.

if (i.examples && i.examples.length>0) {
    	i.examples.map((el,j)=>{
      	result =  el.toLowerCase()
        	.match( self.state.searchQuery.toLowerCase() );
      	if (result) {
          	return result;
      	}
    	})
}

Example

Here the search query is “country”. The word “country” appears the skill description of the filtered cards.

Resources

Adding “All” in Skill Categories

The SUSI SKill CMS has various filters to explore the skills of interest.  For example skill category and skill language. The skills are stored in the susi_skill_data Github repo in the following structure:

susi_skill_data/models/<model_name>/<group_name>/<language_name>/<skill_name>.txt

NOTE: group and category are same terms and can be used interchangeably

So when a category filter is applied the skills from the corresponding directory are returned.

susi_skill_data/models/<model_name>/<group_name>/<language_name>/<skill_name>.txt

But there’s no directory called “All”,  so how to get skills of all groups? For this, we need to loop through all the directories present in the model.

Server side implementation

Create a helper function that returns a list of all the folders present in a directory. The function accepts the parent directory name and an empty list. First, fetch all the items (files and folders) present in that directory and store them in an array. Then apply a filter over the array to check if the element is a directory and doesn’t start with a dot(.) i.e., it’s not hidden. Add the filtered array to the list.

private void listFoldersForFolder(final File folder, ArrayList<String> fileList) {
    File[] filesInFolder = folder.listFiles();
    if (filesInFolder != null) {
        Arrays.stream(filesInFolder)
                .filter(fileEntry -> fileEntry.isDirectory() && !fileEntry.getName().startsWith("."))
                .forEach(fileEntry -> fileList.add(fileEntry.getName() + ""));
    }
}

Fetch the group name form the request and add a check if the CMS is asking for all skill. Otherwise, return the skills of a particular group only.

String group_name = call.get("group", "Knowledge");
if (group_name.equals("All")) {
  // Return the list of all skills
} else {
  // Return the list of a skills in a particular group only
}

To fetch the list of all skills, call the listFoldersForFolders() function with the model name and an empty list as arguments. The function adds all the directories, present in that model directory, to folderList.

File allGroup = new File(String.valueOf(model));
ArrayList<String> folderList = new ArrayList<String>();
listFoldersForFolder(allGroup, folderList);

Then loop over all the groups present in the list to get all the skills present in that group. This process is the same as the existing process of getting skills of a particular category. Just keep adding the skill list to a global array.

CMS side implementation

The list of categories is first fetched from the API and then added to the dropdown menu. Since the API doesn’t return “All” in it, so we need to push it to the list manually.

groups.push(<MenuItem
                value="All"
                key="All"
                primaryText="All" />);

References