Implementing Volume Action in SUSI Smart Speaker

We all know that a Smart Speaker to excel above its competitors has to excel in first being a good “Speaker” and a speaker has a basic and essential feature which is “volume control”. But things get better if you can control your volume with your voice.

So, we have implemented a feature that allows the user to control the volume of the audio with his/her voice.

Below are the steps we had to follow to implement this feature

 

Step 1: Creating the Skills

The skills required to implement the ‘volume-action’ is implemented in the SUSI Server repo itself.

The skill is located in

susi_server/conf/system_skills/general/en/en_0001_foundation.txt

 

And below are the skills required

 

set audio volume to *|set audio volume to * percent|set audio volume to * points|set volume to *|set volume to * percent|set volume to * points
!console:Audio volume is now $1$ percent.
{“actions”:[
{“type”:“audio_volume”, “volume”:$1$}
]}
eol

 

We get the following response from the server

 

“actions”: [
     {
       “volume”: “80”,
       “type”: “audio_volume”
     },
     {
       “type”: “answer”,
       “expression”: “Audio volume is now 80 percent.”
     }

 

Step 2: Finding Volume Action in the server response

Now that our Server responds to our queries regarding the voice change action , we must implement it in our Smart Speaker Client.


We first create a custom class in our in the SUSI API Wrapper repo which has only one member

 

class VolumeAction(BaseAction):
   def __init__(self , volume):
       super().__init__()
       self.volume = volume

 

We check through the actions in the server’s response

 

elif isinstance(action, VolumeAction):
           result[‘volume’] = action.volume

 

Step 3: Implementing it in the client

Now to implement the action in our client we use a library called ‘alsaaudio’ to control the master volume of our RaspberryPi

 

              m = alsaaudio.Mixer()
               m.setvolume(int(reply[‘volume’]))
               os.system(‘play {0} &’.format(self.components.config[‘detection_bell_sound’]))  # nosec #pylint-disable type: ignore                m = alsaaudio.Mixer()
               m.setvolume(int(reply[‘volume’]))
               os.system(‘play {0} &’.format(self.components.config[‘detection_bell_sound’]))  # nosec #pylint-disable type: ignore

 

Now the user can easily change the speaker using the voice commands

References

 

Tags

GSoC, GSoC’18, SUSI.AI, SUSI Linux, Smart Speaker , SUSI API Wrapper, SUSI Server, FOSSASIA, Volume Action

Make an API to check if an email address has been registered for SUSI.AI

This blog post talks about the implementation of the checkRegistration.json API on the SUSI.AI server, which is a part of the AAA system. The API endpoint to check if an email address has been registered for SUSI is https://api.susi.ai/aaa/checkRegistration.json

It accepts one compulsory url parameter –

  • check_email – It is the parameter that contains the string type email address which the user enters in the email address field of the login screen.

The minimalUserRole is set to ANONYMOUS for this API, as initially the registration status of the email address is unknown.

API Development

  • The parameter is first extracted via the post object that is passed to the serviceImpl function. The  parameter is then stored in a variable. If the parameter is absent, then it is set to the default value null.
  • There is a check if the email is null. If null, an exception is thrown.
  • This code snippet discusses the above two points –

@Override
public ServiceResponse serviceImpl(Query post, HttpServletResponse    
   response, Authorization auth, final JsonObjectWithDefault permissions)
			throws APIException {

        String checkEmail = post.get("check_email", null);

        JSONObject result = new JSONObject();
        
        if (checkEmail == null) {
            throw new APIException(422, "Email not provided.");
        }
.
.
.

 

  • Set the credential variable of type ClientCredential by passing the parameters passwd_login and checkEmail to the ClientCredential constructor.
  • Finally pass this credential variable to the getAuthentication method defined in the DAO to return the authentication object.
  • The authentication object then invokes the authentication.getIdentity() method. If the result is null, it means the email address has not been registered yet and vice-versa.
  • Internally, the entire checking procedure is done from the authentication.json file that is stored in data/settings/ directory of the server.
  • The response object is then sent with three key values mainly, apart from the session object. They are
    • accepted –  true – It tells that the API call has been successful.
    • exists – It tells that the email address has already been registered.
    • check_email –  It is the same email address that was sent as a query parameter.

Here are the important code snippets –

  • Continuation of the first code snippet –

.
.
.
// check if id exists already

ClientCredential credential = new    
   ClientCredential(ClientCredential.Type.passwd_login, checkEmail);
	Authentication authentication =DAO.getAuthentication(credential);

		if (authentication.getIdentity() != null) {
			result.put("exists", true);
		} else {
			result.put("exists", false);
		}
		
		result.put("accepted", true);
		result.put("check_email", checkEmail);

		return new ServiceResponse(result);
	}

 

  • Sample response of checkRegistration.json API endpoint –

{
  "check_email": "[email protected]",
  "session": {"identity": {
    "type": "host",
    "name": "127.0.0.1_356778ca",
    "anonymous": true
  }},
  "exists": true,
  "accepted": true
}

 

The API development was done in the above explained way. This API will be used in improving the authentication flow in the Android client, where, if an email address has already been registered, then the user would be taken to the ‘Enter Password Screen’ otherwise he/she would be directed to the Signup screen.

Resources

Fetch Five Star Skill Rating from getSkillList API in SUSI.AI Android

SUSI.AI had a thumbs up/down rating system till now, which has now been replaced by a five star skill rating system. Now, the user is allowed to rate the skill based on a five star rating system. The UI components include a rating bar and below the rating bar is a section that displays the skill rating statistics – total number of ratings, average rating and a graph showing the percentage of users who rated the skill with five stars, four stars and so on.

SUSI.AI Skills are rules that are defined in SUSI Skill Data repo which are basically the processed responses that SUSI returns to the user queries. When a user queries something from the SUSI Android app, a query to SUSI Server is made which in turn fetches data from SUSI Skill Data and returns a JSON response to the app. Similarly, to get skill ratings, a call to the ‘/cms/getSkillList.json’ API is made. In this API, the server checks the SUSI Skill Data repo for the skills and returns a JSON response consisting of all the required information like skill name, author name, description, ratings, etc. to the app. Then, this JSON response is parsed to extract individual fields to display the appropriate information in the skill details screen of the app.

API Information

The endpoint to fetch skills is ‘/cms/getSkillList.json’
The endpoints takes three parameters as input –

  • model – It tells the model to which the skill belongs. The default value is set to general.
  • group – It tells the group(category) to which the skill belongs. The default value is set to All.
  • language – It tells the language to which the skill belongs. The default value is set to en.

Since all skills have to be fetched, this API is called for every group individually. For instance, call “https://api.susi.ai/cms/getSkillList.json?group=Knowledge” to get all skills in group “Knowledge”. Similarly, call for other groups.

Here is a sample response of a skill named ‘Capital’ from the group Knowledge :

"capital": {
      "model": "general",
      "group": "Knowledge",
      "language": "en",
      "developer_privacy_policy": null,
      "descriptions": "A skill to tell user about capital of any country.",
      "image": "images/capital.png",
      "author": "chashmeet singh",
      "author_url": "https://github.com/chashmeetsingh",
      "skill_name": "Capital",
      "terms_of_use": null,
      "dynamic_content": true,
      "examples": ["What is the capital of India?"],
      "skill_rating": {
        "negative": "0",
        "positive": "4",
        "feedback_count" : 0,
        "stars": {
          "one_star": 0,
          "four_star": 1,
          "five_star": 0,
          "total_star": 1,
          "three_star": 0,
          "avg_star": 4,
          "two_star": 0
        }
      },
      "creationTime": "2018-03-17T17:11:59Z",
      "lastAccessTime": "2018-06-06T00:46:22Z",
      "lastModifiedTime": "2018-03-17T17:11:59Z"
    },


It consists of all details about the skill called ‘Capital’:

  1. Model (model)
  2. Group (group)
  3. Language (language)
  4. Developer Privacy Policy (developer_privacy_policy)
  5. Description (descriptions)
  6. Image (image)
  7. Author (author)
  8. Author URL (author_url)
  9. Skill name (skill_name)
  10. Terms of Use (terms_of_use)
  11. Content Type (dynamic_content)
  12. Examples (examples)
  13. Skill Rating (skill_rating)
  14. Creation Time (creationTime)
  15. Last Access Time (lastAccessTime)
  16. Last Modified Time (lastModifiedTime)

From among all this information, the information of interest for this blog is Skill Rating. This blog mainly deals with showing how to parse the JSON response to get the skill rating star values, so as to display the actual data in the skill rating graph.

A request to the getSkillList API is made for each group using the GET method.

@GET("/cms/getSkillList.json")
Call<ListSkillsResponse> fetchListSkills(@Query("group") String groups);

It returns a JSON response consisting of all the aforementioned information. Now, to parse the JSON response, do the following :

  1. Add a response for the response received as a result of API call. ListSkillsResponse contains two objects – group and skills.
    This blog is about getting the skill rating, so let us proceed with parsing the required response. The skills object contains the skill data that we need. Hence, next a SkillData class is created.

    class ListSkillsResponse {
       val group: String = "Knowledge"
       val skillMap: Map<String, SkillData> = HashMap()
    }
  2. Now, add the SkillData class. This class defines the response that we saw for ‘Capital’ skill above. It contains skill name, author, skill rating and so on.

    class SkillData : Serializable {
       var image: String = ""
       @SerializedName("author_url")
       @Expose
       var authorUrl: String = ""
       var examples: List<String> = ArrayList()
       @SerializedName("developer_privacy_policy")
       @Expose
       var developerPrivacyPolicy: String = ""
       var author: String = ""
       @SerializedName("skill_name")
       @Expose
       var skillName: String = ""
       @SerializedName("dynamic_content")
       @Expose
       var dynamicContent: Boolean? = null
       @SerializedName("terms_of_use")
       @Expose
       var termsOfUse: String = ""
       var descriptions: String = ""
       @SerializedName("skill_rating")
       @Expose
       var skillRating: SkillRating? = null
    }
    
  3. Now, add the SkillRating class. As what is required is the skill rating, narrowing down to the skill_rating object. The skill_rating object contains the actual rating for each skill i.e. the stars values. So, this files defines the response for the skill_rating object.

    class SkillRating : Serializable {
       var stars: Stars? = null
    }
    
  4. Further, add a Stars class. Ultimately, the values that are needed are the number of users who rated a skill at five stars, four stars and so on and also the total number of users and the average rating. Thus, this file contains the values inside the ‘stars’ object.

    class Stars : Serializable {
       @SerializedName("one_star")
       @Expose
       var oneStar: String? = null
       @SerializedName("two_star")
       @Expose
       var twoStar: String? = null
       @SerializedName("three_star")
       @Expose
       var threeStar: String? = null
       @SerializedName("four_star")
       @Expose
       var fourStar: String? = null
       @SerializedName("five_star")
       @Expose
       var fiveStar: String? = null
       @SerializedName("total_star")
       @Expose
       var totalStar: String? = null
       @SerializedName("avg_star")
       @Expose
       var averageStar: String? = null
    }
    

Now, the parsing is all done. It is time to use these values to plot the skill rating graph and complete the section displaying the five star skill rating.

To plot these values on the skill rating graph refer to the blog on plotting horizontal bar graph using MPAndroid Chart library. In step 5 of the linked blog, replace the second parameter to the BarEntry constructor by the actual values obtained by parsing.

Here is how we do it.

  • To get the total number of ratings
val  totalNumberofRatings: Int = skillData.skillRating?.stars?.totalStars

 

  • To get the average rating
val averageRating: Float = skillData.skillRating?.stars?.averageStars

 

  • To get number of users who rated the skill at five stars
val fiveStarUsers: Int = skillData.skillRating?.stars?.fiveStar

Similarly, get the number of users for fourStar, threeStar, twoStar and oneStar.

Note : If the totalNumberOfRatings equals to zero, then the skill is unrated. In this case, display a message informing the user that the skill is unrated instead of plotting the graph.

Now, as the graph shows the percentage of users who rated the skill at a particular number of stars, calculate the percentage of users corresponding to each rating, parse the result to Float and place it as the second parameter to the BarEntry constructor  as follows :

 entries.add(BarEntry(4f, (fiveStarUsers!!.toFloat() / totalUsers) * 100f)))

Similarly, replace the values for all five entries. Finally, add the total ratings and average rating section and display the detailed skill rating statistics for each skill, as in the following figure.

Resources

 

 

Upload Avatar for a User in SUSI.AI Server

In this blog post, we are going to discuss on how the feature to upload the avatar for a user was implemented on the SUSI.AI Server. The API endpoint by which a user can upload his/her avatar image is https://api.susi.ai/aaa/uploadAvatar.json.

  • The endpoint is of POST type.
  • It accepts two request parameters –
  • image – It contains the entire image file sent from the client
  • access_token – It is the access_token for the user

The minimalUserRole is set to USER for this API, as only logged-in users can use this API.

Going through the API development

  • The image and access_token parameters are first extracted via the req object, that is passed to the main function. The  parameters are then stored in variables.
  • There is a check if the access_token and image exists. It it doesn’t, an error is thrown.
  • This code snippet discusses the above two points –

protected void doPost(HttpServletRequest req, HttpServletResponse resp) throws ServletException, IOException {
    Part imagePart = req.getPart("image");
    if (req.getParameter("access_token") != null) {
        if (imagePart == null) {
            result.put("accepted", false);
            result.put("message", "Image file not received");
        } else {.
        }
    else{
        result.put("message","Access token are not given");
        result.put("accepted",false);
        resp.setContentType("application/json");
        resp.setCharacterEncoding("UTF-8");
        resp.getWriter().write(result.toString());
    }
}

 

  • Then the input stream is extracted from the imagePart and stored. And post that the identity is checked if it is valid.
  • The input stream is converted into the Image type using the ImageIO.read method.
  • The image is eventually converted into a BufferedImage using a function, described below.

public static BufferedImage toBufferedImage(Image img)
{
    if (img instanceof BufferedImage)
        return (BufferedImage) img;

    // Create a buffered image with transparency
    BufferedImage bimage = new BufferedImage(img.getWidth(null),     
    img.getHeight(null), BufferedImage.TYPE_INT_ARGB);
    // Draw the image on to the buffered image
    Graphics2D bGr = bimage.createGraphics();
    bGr.drawImage(img, 0, 0, null);
    bGr.dispose();
    // Return the buffered image
    return bimage;
}

 

  • After that, the file path and name is set. The avatar for each user is stored in the /data/avatar_uploads/<uuid of the user>.jpg.
  • The avatar is written to the path using the ImageIO.write function. Once, the file is stored on the server, the success response is sent and the client side receives it.

Resources

Overriding the Basic File Attributes while Skill Creation/Editing on Server

In this blog post, we are going to understand the method for overriding basic file attributes while Skill creation/editing on SUSI Server. The need for this arose, when the creationTime for the Skill file that is stored on the server gets changed when the skill was edited.

Need for the implementation

As briefly explained above, the creationTime for the Skill file that is stored on the server gets changed when the skill is edited. Also, the need to override the lastModifiedTime was done, so that the Skills based on metrics gives correct results. Currently, we have two metrics for the SUSI Skills – Recently updated skills and Newest Skills. The former is determined by the lastModifiedTime and the later is determined by the creationTime. Due, to inconsistencies of these attributes, the skills that were shown were out of order. The lastModifiedTime was overridden to save the epoch date during Skill creation, so that the newly created skills don’t show up on the Recently Updated Skills section, whereas the creationTime was overridden to maintain the correct the time.

Going through the implementation

Let us first have a look on how the creationTime was overridden in the ModifySkillService.java file.

.
BasicFileAttributes attr = null;
Path p = Paths.get(skill.getPath());
try {
    attr = Files.readAttributes(p, BasicFileAttributes.class);
} catch (IOException e) {
    e.printStackTrace();
}
FileTime skillCreationTime = null;
if( attr != null ) {
    skillCreationTime = attr.creationTime();
}

if (model_name.equals(modified_model_name) &&
    group_name.equals(modified_group_name) &&
    language_name.equals(modified_language_name) &&
    skill_name.equals(modified_skill_name)) {
    // Writing to File
    try (FileWriter file = new FileWriter(skill)) {
        file.write(content);
        json.put("message", "Skill updated");
        json.put("accepted", true);

    } catch (IOException e) {
        e.printStackTrace();
        json.put("message", "error: " + e.getMessage());
    }
    // Keep the creation time same as previous
    if(attr!=null) {
        try {
            Files.setAttribute(p, "creationTime", skillCreationTime);
        } catch (IOException e) {
            System.err.println("Cannot persist the creation time. " + e);
        }
    }.
}
.
.
.

 

  • Firstly, we get the BasicFileAttributes of the Skill file and store it in the attr variable.
  • Next, we initialise the variable skillCreationTime of type FileTime to null and set the value to the existing creationTime.
  • The new Skill file is saved on the path using the FileWriter instance, which changes the creationTime, lastModifiedTime to the time of editing of the skill.
  • The above behaviour is not desired and hence, we want to override the creationTIme with the FileTime saved in skillCreationTIme. This ensures that the creation time of the skill is persisted, even after editing the skill.
  • Now we are going to see how the lastModifiedTime was overridden in the CreateSkillService.java file.

.
Path newPath = Paths.get(path);
// Override modified date to an older date so that the recently updated metrics works fine
// Set is to the epoch time
try {
  Files.setAttribute(newPath, "lastModifiedTime", FileTime.fromMillis(0));
} catch (IOException e) {
  System.err.println("Cannot override the modified time. " + e);
}
.
.
.

 

  • For this, we get the newPath of the Skill file and then the lastModifiedTime for the Skill File is explicitly set to a particular time.
  • We set it to FileTime.fromMillis(0) i.e, the epoch time.

I hope that I was able to convey my learnings and implementation of the code properly and it proved to be helpful for your understanding.

Resources

Documentation for BasicFileAttributes Interface – https://docs.oracle.com/javase/8/docs/api/java/nio/file/attribute/BasicFileAttributes.html

Adding Filters for Lists of Skills on the SUSI.AI Server

In this blog post, we will learn how to add filters to the API responsible for fetching the list of skills i.e. the endpoint – https://api.susi.ai/cms/getSkillList.json. The purpose of adding filters is to return a list of skills based on some parameters associated with the skill, that would be required to allow the user to get the desired response that s/he may be using to display it on the UI.

Overview of the API

  • API to fetch the list of skills –
    • URL –  https://api.susi.ai/cms/getSkillList.json
    • It takes 5 optional parameters –
      • model – It is the name of the model that user is requesting
      • group – It is the name of the group that user is requesting
      • language – It is the name of the language that user is requesting
      • skill – It is the name of the skill that user is requesting
      • applyFilter – It has true/false values, depending whether filtering is required
    • If the request URL contains the parameter applyFilter as true, in that case the other 2 compulsory parameters are –
      • filter_name – ascending/descending, depending upon the type of sorting the user wants
      • filter_type – lexicographical, rating, etc based on what basis the filtering is going to happen

So, we will now look into adding a new filter_type to the API.

Detailed explanation of the implementation

  • We can add filters based on the key values of the Metadata object of individual skills. The Metadata object for each skill is similar to the following object –

{
  "model": "general",
  "group": "Knowledge",
  "language": "en",
  "developer_privacy_policy": null,
  "descriptions": "A skill that returns the anagrams for a word",
  "image": "images/anagrams.jpg",
  "author": "vivek iyer",
  "author_url": "https://github.com/Remorax",
  "author_email": null,
  "skill_name": "Anagrams",
  "protected": false,
  "terms_of_use": null,
  "dynamic_content": true,
  "examples": ["Anagram for best"],
  "skill_rating": {
    "negative": "0",
    "positive": "0",
    "stars": {
      "one_star": 0,
      "four_star": 0,
      "five_star": 0,
      "total_star": 0,
      "three_star": 0,
      "avg_star": 0,
      "two_star": 0
    },
    "feedback_count": 0
  },
  "creationTime": "2017-12-17T14:32:15Z",
  "lastAccessTime": "2018-06-19T17:50:01Z",
  "lastModifiedTime": "2017-12-17T14:32:15Z"
}

 

  • We will now add provision for URL parameter, filter_type=feedback in the API, which will filter the results based on the feedback_count key, which tells the number of feedback/comments a skill has received.
  • In the serviceImpl method of the ListSkillService class, we can see a code snippet that handles the filtering part, It checks the filter_type parameter received in the URL on if-else block. The code snippet looks like this –

if (filter_type.equals("date")) {
 .
 .
} else if (filter_type.equals("lexicographical")) {
 .
 .
} else if (filter_type.equals("rating")) {
 .
 .
}

 

  • Similarly, we will need to add an else if condition with feedback_type=feedback and write the code block inside it. Here is the code for it, which is explained in detail.

.
.
else if (filter_type.equals("feedback")) {
  if (filter_name.equals("ascending")) {
    Collections.sort(jsonValues, new Comparator<JSONObject>() {

      @Override
      public int compare(JSONObject a, JSONObject b) {
      Integer valA;
      Integer valB;
      int result=0;

      try {
        valA = a.getJSONObject("skill_rating").getInt("feedback_count");
        valB = b.getJSONObject("skill_rating").getInt("feedback_count");
        result = Integer.compare(valA, valB);

      } catch (JSONException e) {
        e.printStackTrace();
      }
      return result;
      }
    });
  }
  else {

    Collections.sort(jsonValues, new Comparator<JSONObject>() {

      @Override
      public int compare(JSONObject a, JSONObject b) {
      Integer valA;
      Integer valB;
      int result=0;

      try {
        valA = a.getJSONObject("skill_rating").getInt("feedback_count");
        valB = b.getJSONObject("skill_rating").getInt("feedback_count");
        result = Integer.compare(valB, valA);
      } catch (JSONException e) {
        e.printStackTrace();
      }
      return result;
    }
  });
  }
}
.
.

Working of the above code snippet

  • The first if condition checks for the filter_type the user has requested and enters the condition if it is equal to feedback
  • The next if-else handles the case of ascending and descending and sorts the list of skills accordingly.
  • The variable jsonValues passed in the Collections.sort function contains a List of JSONObject. Here, each object stands for the metadata object for a skill.
  • Since, the sorting is not a simple linear sort, the sort function is overloaded with a comparator function that specifies the key, based on which the sorting would take place.
  • The feedback_count value is stored in valA and valB for the two variables that is considered at an instance while sorting. For any other key based filtering, we need to replace the feedback_count with the desired key name.
  • The compare() method of Integer class of java.lang package compares two integer values (x, y) given as a parameter and returns the value zero if (x==y), if (x < y) then it returns a value less than zero and if (x > y) then it returns a value greater than zero.
  • The value returned to the sort function determines the order of the sorted array.
  • For, ascending and descending, the parameters of the compare function is swapped, so that we can achieve the exact opposite results from one another.

This was the implementation for the filtering of Skill List based on a key value present in the Skill Metadata object. I hope, you found the blog helpful in making the understanding of the implementation better.

Resources

Server Side Implement of Usage Analytics of a Skill in SUSI.AI

Which skills are being used most frequently in SUSI.AI? And which are rarely used? Such statistics are needed to give proper responses from the most used skills.

So when the user sends a query to SUSI, it searches for the best-suited skill to respond to that query. The usage count of that skill is incremented accordingly with the date.

Storage of Skill Usage Data

  1. Create a skillUsage.json file to store the stats and make a JSONTray object for that in DAO.java file.
  2. Modify the SusiArguement.java file to check which skill is being currently used for the response and write the skill usage stats to skillUsage.json.The function updateUsageData takes the skillPath in parameter and splits it from slash character. Thus the model name, group name, language name and skill name can be obtained. Then it searches for the date in the JSON file. If it exists then its count value is increased by 1, else a new object is inserted in the JSON file that contains the date and the corresponding usage count.

public void updateUsageData(String skillPath) {
    Boolean dateExists = false;
    for (int i = 0; i<usageData.length(); i++) {
        JSONObject dayUsage = usageData.getJSONObject(i);
        if (dayUsage.get("date").equals(today)){
            dayUsage.put("count", dayUsage.getInt("count")+1+"");
            usageData.put(i,dayUsage);
            dateExists = true;
            break;
        }
    }
}   

API to access the Skill Usage Data

  1. Create GetSkillUsageService.java file to return the usage stats stored in skillUsage.json. It runs at the endponit – /cms/getSkillUage.json and mining role required to access this API is anonymous. It simply reads the data stored in skillUsage.json file and returns it.
JsonTray skillRating = DAO.skillUsage;
JSONObject  languageName = groupName.getJSONObject(language_name);
if (languageName.has(skill_name)) {
    JSONArray skillUsage = languageName.getJSONArray(skill_name);
    result.put("skill_name", skill_name);
    result.put("skill_usage", skillUsage);
    result.put("accepted", true);
    result.put("message", "Skill usage fetched");
    return new ServiceResponse(result);
}
  1. Add the API file to skillUsage.json
services = new Class[]{
	...
	//Skill usage data
	GetSkillUsageService.class
	...
}

So now we know which skills are being used at what rate and make the required changes like server scaling, adding more accuracy to the feature etc. We can expand these stats to country wise or device wise usage distribution.

Resources

 

Device wise Usage Statistics of a Skill in SUSI.AI

The device wise usage distribution in SUSI.AI helps in understanding what kind of skills are used more on which type of devices, so that the skill creator can harness the core features of that device to enhance the skills or make the user experience smoother. For example, music playing skill may be used mostly on Smart Speakers whereas Android devices may have higher usage of alarm setting skill.

Sending the device type (ex, web client)

  1. Send the device type parameter as “Web Client” along with the query while fetching reply from SUSI server in chat.json API. The parameter is device_type.

// Add the type of device in the query
{
url += '&device_type=Web Client';
}

Storage of Device Wise Skill Usage Data on SUSI Server

  1. Create a deviceWiseSkillUsage.json file to store the device wise skill usage stats and make a JSONTray object for that in src/ai/susi/DAO.java file. The JSON file contains the device type and the usage count on that type of device (like Android, iOS, Web Client, Smart Speaker and others).
  2. Modify the src/ai/susi/server/api/susi/SusiService.java file to fetch device_type from the query parameters and pass them SusiCognition constructor.

public ServiceResponse serviceImpl(Query post, HttpServletResponse response, Authorization user, final JsonObjectWithDefault permissions) throws APIException {
	...
	String deviceType = post.get("device_type", "Others");
	...
	SusiCognition cognition = new SusiCognition(q, timezoneOffset, latitude, longitude, countryCode, countryName, language, deviceType, count, user.getIdentity(), minds.toArray(new SusiMind[minds.size()]));
	...
}
  1. Modify the src/ai/susi/mind/SusiCognition.java file to accept the deviceType in the constructor parameters. Check which skill is being currently used for the response and update the skill’s usage stats for the current device in deviceWiseSkillUsage.json. Call the function updateDeviceWiseUsageData() to update the skill usage data.

List<String> skills = dispute.get(0).getSkills();
for (String skill : skills) {
    updateDeviceWiseUsageData(skill, deviceType);
}

The updateDeviceWiseUsageData() function accepts the skill path and type of device. It parses the skill path to get the skill metadata like its model name, group name, language etc. The function then checks if the device already exists in the JSON file or not. If it exists then it increments the usage count by 1 else it creates an entry for the device in the JSON file and initializes it with the usage count 1.

for (int i = 0; i < deviceWiseUsageData.length(); i++) {
  deviceUsage = deviceWiseUsageData.getJSONObject(i);
  if (deviceUsage.get("device_type").equals(deviceType)) {
    deviceUsage.put("count", deviceUsage.getInt("count") + 1);
    deviceWiseUsageData.put(i,deviceUsage);
  }
}

API to access the Device Wise Skill Usage Data

  1. Create GetDeviceWiseSkillUsageService.java file to return the usage stats stored in deviceWiseSkillUsage.json

public ServiceResponse serviceImpl(Query call, HttpServletResponse response, Authorization rights, final JsonObjectWithDefault permissions) {        
  ...  // Fetch the query parameters
  JSONArray deviceWiseSkillUsage = languageName.getJSONArray(skill_name);
  result.put("skill_name", skill_name);
  result.put("skill_usage", deviceWiseSkillUsage);
  result.put("accepted", true);
  result.put("message", "Device wise skill usage fetched"); 
  return new ServiceResponse(result);    
}
  1. Add the API file to src/ai/susi/server/api/susi/SusiServer.java

services = new Class[]{
	...

	//Skill usage data
	GetDeviceWiseSkillUsageService.class
	
	...
}

 

Endpoint : /cms/getDeviceWiseSkillUsage.json

Parameters

  • model
  • group
  • language
  • Skill

Sample query: /cms/getDeviceWiseSkillUsage.json?model=general&group=Knowledge&language=en&skill=aboutsusi

Sample response:

{  
   "skill_usage":[  
    {
      "device_type": "Web Client",
      "count": 1
    },
    {
        "device_type": "Android",
        "count": 4
    },
    {
        "device_type": "iOS",
        "count": 2
    },
    {
        "device_type": "Smart Speaker",
        "count": 1
    },
    {
        "device_type": "Others",
        "count": 2
    }
   ],
   "session":{  
      "identity":{  
         "type":"host",
         "name":"162.158.154.147_81c88a10",
         "anonymous":true
      }
   },
   "skill_name":"ceo",
   "accepted":true,
   "message":"Device wise skill usage fetched"
}

Resources

Fetching Bots from SUSI.AI Server

Public skills of SUSI.AI are stored in susi_skill_cms repository. A private skill is different from a public skill. It can be viewed, edited and deleted only by the user of who created it. The purpose of private skill is that this acts as a chatbot. All the information of the bot like design, configuration etc is stored within the private skill itself. In order to show the lists of chatbots a user has created to him/her, we need to fetch these lists from the server. The API for fetching private skills is in ListSkillService.java servlet. This servlet is used for both the purposes for fetching public skills and private skills.

How are bots fetched?

All the private skills are stored in data/chatbot/chatbot.json location in the SUSI server. Hence, we can fetch these skills from here.
The API call to fetch private skills is:

https://api.susi.ai/cms/getSkillList.json?private=1&access_token=accessTokenHere

The API call has two parameters:

  1. private = 1: This parameter tells the servlet that we’re asking for private skills because same API is used for fetching public skills as well.
  2. access_token : We have to pass the access-token in order to authenticate the user and access the chatbots of the specific user because the chatbots are stored according to user id of the users in chatbot.json. To get the user id, we need access-token.

For fetching the chatbots from chatbot.json, we firstly authenticate the user. Then we check if the user has any chatbots or not. If the user has chatbots then we loop through chatbot.json to find the chatbots of user by using user id. After getting chatbots details, we simply present it in json structure. You can easily understand this through the following code snippets. For authenticating:

String userId = null;
String privateSkill = call.get("private", null);
if (call.get("access_token", null) != null) {
    ClientCredential credential = new ClientCredential(ClientCredential.Type.access_token, call.get("access_token", null));
    Authentication authentication = DAO.getAuthentication(credential);
    // check if access_token is valid
    if (authentication.getIdentity() != null) {
        ClientIdentity identity = authentication.getIdentity();
        userId = identity.getUuid();
    }
}

You can see that the parameter private is stored in privateSkill. We check if privateSkill is not null followed by checking if the user id provided is valid or not. When both the checks are successful, we create a JSON object in which we store all the bots. To do this, we loop through the chatbot.json file to find the user id provided in API call. If the user id doesn’t exist in chatbot.json file then a message “User has no chatbots.” is displayed. If the user id is found then we again loop through all the chatbots and put their name as the value of key “name”,  language as the value of key “language” and group as the value of key “group”. All these key value pairs for the chatbots are pushed in an array and finally that array is added to the JSON object. This JSON is then received as response of the API call. The following code will demonstrate this:

if(privateSkill != null) {
    if(userId != null) {
        JsonTray chatbot = DAO.chatbot;
        JSONObject result = new JSONObject();
        JSONObject userObject = new JSONObject();
        JSONArray botDetailsArray = new JSONArray();
        JSONArray chatbotArray = new JSONArray();
        for(String user_id : chatbot.keys())
        {
            if(user_id.equals(userId)) {
                userObject = chatbot.getJSONObject(user_id);
                Iterator chatbotDetails = userObject.keys();
                List<String> chatbotDetailsKeysList = new ArrayList<String>();
                while(chatbotDetails.hasNext()) {
                    String key = (String) chatbotDetails.next();
                    chatbotDetailsKeysList.add(key);
                }
                for(String chatbot_name : chatbotDetailsKeysList)
                {
                    chatbotArray = userObject.getJSONArray(chatbot_name);
                    for(int i=0; i<chatbotArray.length(); i++) {
                        String name = chatbotArray.getJSONObject(i).get("name").toString();
                        String language = chatbotArray.getJSONObject(i).get("language").toString();
                        String group = chatbotArray.getJSONObject(i).get("group").toString();
                        JSONObject botDetails = new JSONObject();
                        botDetails.put("name", name);
                        botDetails.put("language", language);
                        botDetails.put("group", group);
                        botDetailsArray.put(botDetails);
                        result.put("chatbots", botDetailsArray);
                    }
                }
            }
        }
        if(result.length()==0) {
            result.put("accepted", false);
            result.put("message", "User has no chatbots.");
            return new ServiceResponse(result);
        }
        result.put("accepted", true);
        result.put("message", "All chatbots of user fetched.");
        return new ServiceResponse(result);
    }
}

So, if chatbot.json contains:

{"c9b58e182ce6466e413d5acafae906ad": {"chatbots": [
 {
   "name": "testing111",
   "language": "en",
   "group": "Knowledge"
 },
 {
   "name": "DNS_Bot",
   "language": "en",
   "group": "Knowledge"
 }
]}}

Then, the JSON received at http://api.susi.ai/cms/getSkillList.json?private=1&access_token=JwTlO8gdCJUns569hzC3ujdhzbiF6I is:

{
  "session": {"identity": {
    "type": "email",
    "name": "[email protected]",
    "anonymous": false
  }},
  "chatbots": [
    {
      "name": "testing111",
      "language": "en",
      "group": "Knowledge"
    },
    {
      "name": "DNS_Bot",
      "language": "en",
      "group": "Knowledge"
    }
  ],
  "accepted": true,
  "message": "All chatbots of user fetched."
}

References:

Countrywise Usage Analytics of a Skill in SUSI.AI

The statistics regarding which country the skills are being used is quite important. They help in updating the skill to support the native language of those countries. SUSI.AI must be able to understand as well as reply in its user’s language. So mainly the server side and some client side (web client) implementation of country wise skill usage statistics is explained in this blog.

Fetching the user’s location on the web client

  1. Add a function in chat.susi.ai/src/actions/API.actions.js to fetch the users location. The function makes a call to freegeoip.net API which returns the client’s location based on its IP address. So country name and code are required for country wise usage analytics.

export function getLocation(){
  $.ajax({
    url: 'https://cors-anywhere.herokuapp.com/http://freegeoip.net/json/',
    success: function (response) {
      _Location = {
        lat: response.latitude,
        lng: response.longitude,
        countryCode: response.country_code,
        countryName: response.country_name
      };
    },
  });
}
  1. Send the location parameters along with the query while fetching reply from SUSI server in chat.json API. The parameters are country_name and country_code.

if(_Location){
	url += '&latitude='+_Location.lat+'&longitude='+_Location.lng+'&country_code='+_Location.countryCode+'&country_name='+_Location.countryName;
}

Storage of Country Wise Skill Usage Data on SUSI Server

  1. Create a countryWiseSkillUsage.json file to store the country wise skill usage stats and make a JSONTray object for that in src/ai/susi/DAO.java file. The JSON file contains the country name, country code and the usage count in that country.
  1. Modify the src/ai/susi/server/api/susi/SusiService.java file to fetch country_name and country_code from the query parameters and pass them SusiCognition constructor.

String countryName = post.get("country_name", "");
String countryCode = post.get("country_code", "");
...

SusiCognition cognition = new SusiCognition(q, timezoneOffset, latitude, longitude, countryCode, countryName, language, count, user.getIdentity(), minds.toArray(new SusiMind[minds.size()]));
  1. Modify the src/ai/susi/mind/SusiCognition.java file to accept the countryCode and countryName in the constructor parameters. Check which skill is being currently used for the response and update the skill usage stats for that country in countryWiseSkillUsage.json. Call the function updateCountryWiseUsageData() to update the skill usage data.

if (!countryCode.equals("") && !countryName.equals("")) {
    List<String> skills = dispute.get(0).getSkills();
    for (String skill : skills) {
        try {
            updateCountryWiseUsageData(skill, countryCode, countryName);
        }
        catch (Exception e) {
            e.printStackTrace();
        }
    }
}

The updateCountryWiseUsageData() function accepts the skill path , country name and country code. It parses the skill path to get the skill metadata like its model name, group name, language etc. The function then checks if the country already exists in the JSON file or not. If it exists then it increments the usage count by 1 else it creates an entry for the skill in the JSON file and initializes it with the current country name and usage count 1.

for (int i = 0; i < countryWiseUsageData.length(); i++) {
  countryUsage = countryWiseUsageData.getJSONObject(i);
  if (countryUsage.get("country_code").equals(countryCode)) {
    countryUsage.put("count", countryUsage.getInt("count")+1);
    countryWiseUsageData.put(i,countryUsage);
  }
}

API to access the Country Wise Skill Usage Data

  1. Create GetCountryWiseSkillUsageService.java file to return the usage stats stored in countryWiseSkillUsage.json

public ServiceResponse serviceImpl(Query call, HttpServletResponse response, Authorization rights, final JsonObjectWithDefault permissions) {
        
  ...  // Fetch the query parameters
  JSONArray countryWiseSkillUsage = languageName.getJSONArray(skill_name);
  return new ServiceResponse(result);
}
  1. Add the API file to src/ai/susi/server/api/susi/SusiServer.java

services = new Class[]{
	...
	//Skill usage data
	GetCountryWiseSkillUsageService.class
	...
}

 

Endpoint : /cms/getCountryWiseSkillUsage.json

Parameters

  • model
  • group
  • language
  • Skill

Sample query: /cms/getCountryWiseSkillUsage.json?model=general&group=Knowledge&language=en&skill=aboutsusi

Sample response:

{  
   "skill_usage":[  
      {  
         "country_code":"MYS",
         "country_name":"Malaysia",
         "count":1
      },
      {  
         "country_code":"MYS",
         "country_name":"Malaysia",
         "count":1
      }
   ],
   "session":{  
      "identity":{  
         "type":"host",
         "name":"162.158.154.147_81c88a10",
         "anonymous":true
      }
   },
   "skill_name":"ceo",
   "accepted":true,
   "message":"Country wise skill usage fetched"
}

Resources