Creating Feedback Logs for Analysis

The thumbs up and thumbs down feedback on the clients is meant for the improvement of the skills in SUSI.AI. So we need to scope the feedback system to a particular interaction rather than skill as a whole. The feedback logs can be used for various kinds of analysis and machine learning.

Server side implementation

Components of Feedback Log:

  • User ID – For identification of a feedback given by a particular user. For consistency in data, the user should not be able to change the feedback over the same interaction.
  • Interaction:
    • User query
    • SUSI Reply
  • Client location – The response of a skill may not be interesting for the users of a particular country. That means the skill should give localised results.
  • Skill path – The path on the server where the skill is stored.

Create a feedbackLogs.json file to store the logs of feedback given by the user and make a JSONTray object for that in src/ai/susi/DAO.java file. The JSON file contains the above mentioned components.

public static JsonTray feedbackLogs; 

Path feedbackLogs_per = susi_skill_rating_dir.resolve("feedbackLogs.json");
Path feedbackLogs_vol = susi_skill_rating_dir.resolve("feedbackLogs_session.json");
feedbackLogs = new JsonTray(feedbackLogs_per.toFile(), feedbackLogs_vol.toFile(), 1000000);
OS.protectPath(feedbackLogs_per);
OS.protectPath(feedbackLogs_vol);

Create FeedbackLogService.java file that acts as an API to create the feedback logs. The API accepts the feedback data from the client and stores it into the json file using DAO object. The user should be logged in to give feedback on an interaction. So keep the minimum user role as USER to access the API.

JSONObject feedbackLogObject = new JSONObject();
feedbackLogObject.put("timestamp", timestamp);
feedbackLogObject.put("uuid", idvalue);
feedbackLogObject.put("feedback", skill_rate);
feedbackLogObject.put("user_query", user_query);
feedbackLogObject.put("susi_reply", susi_reply);
feedbackLogObject.put("country_name", country_name);
feedbackLogObject.put("country_code", country_code);
feedbackLogObject.put("skill_path", skill_path);

The API is accessible at /cms/feedbackLog.json endpoint.

Send feedback log from Web Client

The feedback API should be called only if the user is logged in. When the user presses the feedback buttons fetch the required data for log (access token, user query, susi response, country and user feedback) and POST them on the feedbackLog.json API.

let rateEndPoint =   BASE_URL + '/cms/feedbackLog.json?model=' + skill.model + '&group=' + skill.group + '&language=' + skill.language + '&skill=' + skill.skill + '&rating=' + rating + '&access_token=' + accessToken + '&user_query=' + interaction.userQuery + '&susi_reply=' + interaction.susiReply + '&country_name=' + country.countryName + '&country_code=' + country.countryCode ;

$.ajax({
  url: rateEndPoint,
  success: function(response) {
      console.log('Skill rated successfully');
  }
})

References

Continue ReadingCreating Feedback Logs for Analysis

Showing only those languages for which skills are available

SUSI.AI is available for almost all the internationally recognised languages of the world. An author is allowed to create a skill in any of these languages. But there are some languages for which skills have not been created yet. So only those languages should be shown in the SUSI Skill CMS for which the skills are available. The approach is that all the languages must be listed while creating the skills but only non-empty languages must be listed while filtering skills on the CMS category page.

Updating the get languages API

  1. Add an API parameter in GetAllLanguages.java to fetch the group name. It is used to fetch the list of languages for which skills are available in that particular group. If no group is passed it means that the all the languages are to be listed. For that, we can use any group and show all the languages in that group. Say “Knowledge”.

String group_name = call.get("group", null);
if (group_name == null) {
    File group = new File(model, "Knowledge");
}

 

  1. Now check if the file inside the group folder is a directory. If yes then add it to the list of languages to be returned.

String[] languages = group.list((current, name) -> new File(current, name).isDirectory());

 

  1. If the languages corresponding to a particular category are to be fetched then first checked if the group is “All” or any specific group. Since the “All” category is not stored as such so we need to iterate over all the groups present in the parent directory ie, the model directory.

String[] group_names = model.list((current, name) -> new File(current, name).isDirectory());

 

  1. Now iterate over all the groups present in the group_names array and list the files present in it. Apply a filter to the list that accepts a file only if it is a directory and not empty ie, contains at least 1 language. Add that file to the list of languages.

group.list(new FilenameFilter() {
	@Override
	public boolean accept(File file, String s) {
		Boolean accepted = new File(file, s).list().length > 1;
		if (accepted) {
			if (!languages.contains(s)) {
		    	languages.add(s);
			}
		}
		return accepted;
	}
});

 

  1. The processing of getting languages for a particular group is same, just the iteration over the model directory is not required.

Resources

Continue ReadingShowing only those languages for which skills are available

SUSI AI 5 Star Skill Rating System

For making a system more reliable and robust, continuous evaluation is quite important. So is in case of SUSI AI. User feedback is important to improve SUSI skills and create new ones. Previously we had only thumbs up / thumbs down as a feedback method, from the susi chat client. But now a 5 star rating system has been added to the SUSI Skill CMS so that users can rate a skill there. Before the implementation of API  let’s look how data is stored in SUSI AI Susi_server uses DAO in which skill rating is stored as JSONTray.

The server side implementation

A new java class has been created for the API, FiveStarRateSkillService.java.

public class FiveStarRateSkillService extends AbstractAPIHandler implements APIHandler {
    private static final long serialVersionUID =7947060716231250102L;
    @Override
    public BaseUserRole getMinimalBaseUserRole() {
        return BaseUserRole.USER;
    }
    @Override
    public JSONObject getDefaultPermissions(BaseUserRole baseUserRole) {
        return null;
    }
    @Override
    public String getAPIPath() {
        return "/cms/rateSkill.json";
    }
...
}

The getMinimalBaseRole method tells the minimum User role required to access this servlet it can also be ADMIN, USER or ANONYMOUS. In our case it is USER. A user needs to be logged in to rate a skill on a scale of 1-5 stars.  The API runs at “/cms/fiveStarRateSkill.json” endpoint.

Next, create serviceImpl method in the above class to handle the request from the client and respond to it.

1. Fetch the required query parameters and store them in variables. They include skill model, group, language, skill name and starts that the user has given in the rating.

String skill_name = call.get("skill", null);
String skill_stars = call.get("stars", null);

2. Then check if the skill exists. If not them throw an exception. Otherwise, increment the count of the corresponding rating. The rating object has keys as one_star, two_star, three_star, four_star and five_star that has the count of that star rating.       

if (skill_stars.equals("1")) {
    skillName.put("one_star", skillName.getInt("one_star") + 1 + "");
}
else if (skill_stars.equals("2")) {
    skillName.put("two_star", skillName.getInt("two_star") + 1 + "");
}
else if (skill_stars.equals("3")) {
    skillName.put("three_star", skillName.getInt("three_star") + 1 + "");
}
else if (skill_stars.equals("4")) {
    skillName.put("four_star", skillName.getInt("four_star") + 1 + "");
}
else if (skill_stars.equals("5")) {
    skillName.put("five_star", skillName.getInt("five_star") + 1 + "");
}

3. Re-calculate the total rating done on that skill and its average rating and update the object. If the skill has not been already rated then create a new rating object and initialize it with the 0 star counts.

public JSONObject createRatingObject(String skill_stars) {
        JSONObject skillName = new JSONObject();
        JSONObject skillStars = new JSONObject();

        skillStars.put("one_star", 0);
        skillStars.put("two_star", 0);
        skillStars.put("three_star", 0);
        skillStars.put("four_star", 0);
        skillStars.put("five_star", 0);
        skillStars.put("avg_star", 0);
        skillStars.put("total_star", 0);

        skillName.put("stars", skillStars);
}

The complete FiveStarRateSkillService.java is available here : –

https://github.com/fossasia/susi_server/blob/development/src/ai/susi/server/api/cms/FiveStarRateSkillService.java

Rating a skill

Sample endpoint

https://api.susi.ai/cms/fiveStarRateSkill.json?model=general&group=Knowledge&language=en&skill=aboutsusi&stars=3&callback=p&_=1526813916145

This gives 3 star rating to the “aboutsusi” skill.

Parameters

  • Model
  • Group
  • Language
  • Skill
  • Stars

Response

{
  "ratings": {
    "one_star": 0,
    "four_star": 0,
    "five_star": 1,
    "total_star": 1,
    "three_star": 0,
    "avg_star": 5,
    "two_star": 0
  },
  "session": {"identity": {
    "type": "email",
    "name": "1anuppanwar@gmail.com",
    "anonymous": false
  }},
  "accepted": true,
  "message": "Skill ratings updated"
}

Getting the stats of Skill Ratings

Sample endpoint

http://127.0.0.1:4000/cms/getSkillRating.json?model=general&group=Knowledge&language=en&skill=aboutsusi&callback=p&_=1526813916145

This fetches the current ratings of the “aboutsusi” skill.

Parameters

  • Model
  • Group
  • Language
  • Skill

Response

{
    "session": {
        "identity": {
            "type": "host",
            "name": "172.68.144.159_81c88a10",
            "anonymous": true
        }
    },
    "skill_name": "aboutsusi",
    "accepted": true,
    "message": "Skill ratings fetched",
    "skill_rating": {
        "negative": "0",
        "positive": "0",
        "stars": {
            "one_star": 0,
            "four_star": 2,
            "five_star": 1,
            "total_star": 4,
            "three_star": 1,
            "avg_star": 4,
            "two_star": 0
        },
        "feedback_count": 3
    }
}

Conclusion

So this 5 star rating system will help in improving the SUSI skills. Also, it will help in making better decisions when we have multiple similar skills and we have to choose one to respond to the user query.

References

Continue ReadingSUSI AI 5 Star Skill Rating System

How search works on Susi Skill CMS

The SUSI Skill CMS is a central dashboard where the developers can add skills to make SUSI more intelligent. One of the major thing that was missing there was a search bar. So I recently added one that can search a skill based on:

  • Skill name
  • Skill description
  • Author name
  • Examples of the skill

How to add the search bar?

  1. Install the Material Search Bar component from the terminal.

npm i --save material-ui-search-bar

 

  1. Import this component in the BrowseSkill.js file.

import SearchBar from 'material-ui-search-bar'

 

  1. Create a state variables for the search query and initialize it to an empty string.

this.state = {
	...
	searchQuery:''
	...
};

 

  1. Add the search bar (UI Part) below the filters on CMS. Add a listener function to it that is called when the value of the search query changes.

<SearchBar
     onChange={this.handleSearch}
     value={this.state.searchQuery}
   />

 

  1. The search handler : Create a handler (handleSearch) to that listens to onChange events on the SearchBar. This function sets the value of searhQuery state variable and loads the card again based on the search filter.

handleSearch
  = (value) => {
    this.setState({searchQuery: value}, function () {
    this.loadCards();
    });
};

 

  1. The loadCards() function : The loadCards() function send a request to the Susi Server which in turn send a json response. Then this function makes cards for every skill and adds them to the CMS. Modify the loadCards() function to filter the cards array based on the search query. The javascript string function match() is used to check if the skill name, description, author’s name or examples match the search query. The filter() function adds the skill card to the filtered data if the match() function returns true (i.e., the skill is relevant to the search query).

How the filter works?

  1. First check if there’s something to search for. If the searchQuery.length is equal to zero then that means that there is nothing to search for.

self.state.searchQuery.length >0

 

  1. Then filter the related results based on
  • Skill name : The filter() function adds the skill card to the filtered data if the match function returns true (i.e., the skill is relevant to the search query). The match() function retuns true if the skill name matches the search query.

if (i.skill_name) {
    	result =  i.skill_name.toLowerCase()
        	.match( self.state.searchQuery.toLowerCase() );
    	if (result) {
        	return result;
    	}
}

 

Similarly, filter the cards on the basis of skill description and skill author’s name.

  • Skill examples: Loop over all the skill examples and check if any example matches the search query.

if (i.examples && i.examples.length>0) {
    	i.examples.map((el,j)=>{
      	result =  el.toLowerCase()
        	.match( self.state.searchQuery.toLowerCase() );
      	if (result) {
          	return result;
      	}
    	})
}

Example

Here the search query is “country”. The word “country” appears the skill description of the filtered cards.

Resources

Continue ReadingHow search works on Susi Skill CMS

Adding “All” in Skill Categories

The SUSI SKill CMS has various filters to explore the skills of interest.  For example skill category and skill language. The skills are stored in the susi_skill_data Github repo in the following structure:

susi_skill_data/models/<model_name>/<group_name>/<language_name>/<skill_name>.txt

NOTE: group and category are same terms and can be used interchangeably

So when a category filter is applied the skills from the corresponding directory are returned.

susi_skill_data/models/<model_name>/<group_name>/<language_name>/<skill_name>.txt

But there’s no directory called “All”,  so how to get skills of all groups? For this, we need to loop through all the directories present in the model.

Server side implementation

Create a helper function that returns a list of all the folders present in a directory. The function accepts the parent directory name and an empty list. First, fetch all the items (files and folders) present in that directory and store them in an array. Then apply a filter over the array to check if the element is a directory and doesn’t start with a dot(.) i.e., it’s not hidden. Add the filtered array to the list.

private void listFoldersForFolder(final File folder, ArrayList<String> fileList) {
    File[] filesInFolder = folder.listFiles();
    if (filesInFolder != null) {
        Arrays.stream(filesInFolder)
                .filter(fileEntry -> fileEntry.isDirectory() && !fileEntry.getName().startsWith("."))
                .forEach(fileEntry -> fileList.add(fileEntry.getName() + ""));
    }
}

Fetch the group name form the request and add a check if the CMS is asking for all skill. Otherwise, return the skills of a particular group only.

String group_name = call.get("group", "Knowledge");
if (group_name.equals("All")) {
  // Return the list of all skills
} else {
  // Return the list of a skills in a particular group only
}

To fetch the list of all skills, call the listFoldersForFolders() function with the model name and an empty list as arguments. The function adds all the directories, present in that model directory, to folderList.

File allGroup = new File(String.valueOf(model));
ArrayList<String> folderList = new ArrayList<String>();
listFoldersForFolder(allGroup, folderList);

Then loop over all the groups present in the list to get all the skills present in that group. This process is the same as the existing process of getting skills of a particular category. Just keep adding the skill list to a global array.

CMS side implementation

The list of categories is first fetched from the API and then added to the dropdown menu. Since the API doesn’t return “All” in it, so we need to push it to the list manually.

groups.push(<MenuItem
                value="All"
                key="All"
                primaryText="All" />);

References

Continue ReadingAdding “All” in Skill Categories