Adding Description to the Susi AI Skills

Susi skill CMS is an editor to write and edit skill easily. It follows an API-centric approach where the Susi server acts as API server and a web front-end act as the client for the API and provides the user interface. A skill is a set of intents. One text file represents one skill, it may contain several intents which all belong together. All the skills are stored in Susi Skill Data repository and the schema is as following.

Using this, one can access any skill based on four tuples parameters model, group, language, skill. To know what a skill is about we needed to add a !description operator which identifies the text as a description for the skill. Let’s check out how to achieve it.Susi Skill class provides parser methods for the set of intents, given as text files.

 public static JSONObject readEzDSkill(BufferedReader br) throws JSONException {}
if (line.startsWith("!") && (thenpos = line.indexOf(':')) > 0) {
        String head = line.substring(1, thenpos).trim().toLowerCase();
       String tail = line.substring(thenpos + 1).trim();
if (head.equals("description")) {
   description =tail;
    }
}
 if (description.length() > 0) intent.put("description", description); 

The method readEzDSkill parses the skill txt file, it checks if a line starts with ‘!description’ (‘bang operator with description’) it then stores the content in string variable description.
If a description is found in a skill, it is recorded and put into Json Array of intents.

private final Map<String, Set<String>> skillDescriptions; 
 if (intent.getDescription() !=null) {
  Set<String> descriptions = this.skillDescriptions.get(intent.getSkill());
  if (descriptions == null) {
     descriptions = new LinkedHashSet<>();
     this.skillDescriptions.put(intent.getSkill(), descriptions);
   }
   descriptions.add(intent.getDescription());
}

SusiMind class  process this json and stores the description in a map of skill path and description. This map is used by DescriptionSkillService to list descriptions for all the skills given its model, group and language. For adding the description servlet we need to inherit the service class from AbstractAPIHandler and implement APIhandler interface.In Susi Server, an abstract class AbstractAPIHandler extending HttpServelets and implementing API handler interface is provided.

 @Override
    public BaseUserRole getMinimalBaseUserRole() { return BaseUserRole.ANONYMOUS; }

    @Override
    public JSONObject getDefaultPermissions(BaseUserRole baseUserRole) {
        return null;
    }

    @Override
    public String getAPIPath() {
        return "/cms/getDescriptionSkill.json";
    }

The getAPIPath() methods sets the API endpoint path, it gets appended to base path which is 127.0.0.1:4000/cms/getDescriptionSkill.json for local host. The getMinimalBaseRole method tells the minimum Userrole required to access this servlet it can also be ADMIN, USER. In our case it is Anonymous. A User need not log in to access this endpoint.
Next, we implement serviceimpl method which gives us the desired response in JSON format.

@Override
    public ServiceResponse serviceImpl(Query call, HttpServletResponse response, Authorization rights, final JsonObjectWithDefault permissions) {
        String model = call.get("model", "");
        String group = call.get("group", "");
        String language = call.get("language", "");
        JSONObject descriptions = new JSONObject(true);
            for (Map.Entry<String, Set<String>> entry : DAO.susi.getSkillDescriptions().entrySet()) {
                String path = entry.getKey();
  if ((model.length() == 0 || path.indexOf("/" + model + "/") > 0) &&(group.length() == 0 || path.indexOf("/" + group + "/") > 0) &&(language.length() == 0 || path.indexOf("/" + language + "/") > 0)) {
      descriptions.put(path, entry.getValue());
   }
            }
            JSONObject json = new JSONObject(true)
                    .put("model", model)
                    .put("group", group)
                    .put("language", language)
                    .put("descriptions", descriptions);
        return new ServiceResponse(json);
    }

We can get the required parameters through a call.get() method where the first parameter is the key for which we want to get the value and second parameter is the default value. If the path contains the desired language, group and model, we return it as a response otherwise an error message is displayed. To check the response go to http://api.susi.ai/cms/getDescriptionSkill.json?model=general&group=knowledge&language=en or http://127.0.0.1:4000/cms/getDescriptionSkill.json.

This is how getDescriptionSkill service works. To add a description to the skill visit susi_skill_data, the storage place for susi skills. For more information and complete code take a look at Susi server and join gitter chat channel for discussions.

Resources

Skill History Component in Susi Skill CMS

SUSI Skill CMS is an editor to write and edit skill easily. It is built on ReactJS framework and follows an API centric approach where the Susi server acts as API server. Using Skill CMS we can browse history of a skill, where we get commit ID, commit message and name the author who made the changes to that skills. In this blog post, we will see how to add skill revision history component in Susi Skill CMS.

One text file represents one skill, it may contain several intents which all belong together. Susi skills are stored in susi_skill_data repository. We can access any skill based on four tuples parameters model, group, language, skill.

<Menu.Item key="BrowseRevision">
 <Icon type="fork" />
   Browse Skills Revision
 <Link to="/browseHistory"></Link>
</Menu.Item>

First let’s create a option in sidebar menu, and link it “/browseHistory” route created at index.js 

 <Route path="/browseHistory" component={BrowseHistory} />

Next we will be adding skill versioning using endpoints provided by Susi Server, to select a skill we will create a drop down list, for this we will be using Select Field a component of  Material UI.

request('http://cors-anywhere.herokuapp.com/api.susi.ai/cms/getModel.json').then((data) => {
    console.log(data.data);
    data = data.data;
    for (let i = 0; i < data.length; i++) {
        models.push(<MenuItem value={i} key={data[i]} primaryText={`${data[i]}`}/>);
    }

    console.log(models);
});
 <SelectField
   floatingLabelText="Expert"
   style={{width: '50px'}}
   value={this.state.value}
   onChange={this.handleChange}>
      {experts}
  </SelectField>

We store the models, groups and languages in array using the endpoints api.susi.ai/cms/getModel.json, api.susi.ai/cms/getGroups.json, api.susi.ai/cms/getAllLanguages.json set the values in respective select fields. The request functions takes the url as string and the parses the json and fetches the object containing data or error depending on the response from the server. Once run your project using

npm start

And you would be able to see the drop down list working

Next, we will use Material UI tables for displaying the organized data. For using  table component we need to import table, it’s body, header and row column from Material ui class. 

import {Table, TableBody, TableHeader, TableHeaderColumn, TableRow, TableRowColumn} from "material-ui/Table";

We then make our header Columns, in our case it’s three, namely Commit ID, Commit Message, Author name.

 <TableRow>
  <TableHeaderColumn tooltip="Commit ID">Commit ID</TableHeaderColumn>
  <TableHeaderColumn tooltip="Commit Message">Commit Message</TableHeaderColumn>
  <TableHeaderColumn tooltip="Author Name">Author Name</TableHeaderColumn>
 </TableRow>

To get the history of modification of a skill, we will use endpoint “http://api.susi.ai/cms/getSkillHistory.json”. It uses JGit for managing version control in skill data repository. JGit is a library which implements the Git functionality in Java. An endpoint is accessed based on userRoles, which can be Admin, Privilege, User, Anonymous. In our case it is Anonymous. Thus a User need not to log in to access this endpoint.

 url = "http://api.susi.ai/cms/getSkillHistory.json?model="+models[this.state.modelValue].key+"&group="+groups[this.state.groupValue].key+"&language="+languages[this.state.languageValue].key+"&skill="+this.state.expertValue;

After getting the url, we will next make a ajax network call to get the modification history of skill., if the method returns success, we get the desired data in table array and display it in rows through its render() method, which checks if the data is set in state -if so, it renders the contents otherwise we display the error occurred while processing the request.

 $.ajax({
            url: url,
            jsonpCallback: 'pccd',
            dataType: 'jsonp',
            jsonp: 'callback',
            crossDomain: true,
            success: function(data) {
                data = data.commits;
                let array = [];
                for(let i=0;i<data.length;i++){
                    array.push(data[i]);
                }
{tableData.map((row, index) => (
  <TableRow key={index}>
    <TableRowColumn>{row.commitID}</TableRowColumn>
    <TableRowColumn>{row.commit_message}</TableRowColumn>
    <TableRowColumn>{row.author}</TableRowColumn>
   </TableRow>
))}

Test the final output  on http://skills.susi.ai/browseHistory or http://localhost:3000/browseHistory , select the model, group , language and skill and get the history of that skill.


Next time when you need drop down list or tables to organize your data, do check out https://github.com/callemall/material-ui for examples, which can help in providing good outlines to you apps. For contributions to susi_skill_cms, join our chat channel  on gitter: https://gitter.im/fossasia/susi_server and browse https://github.com/fossasia/susi_skill_cms for complete code.

Resources

Authentication in SUSI.AI

Authentication is a part of AAA system which stands for Authentication, Authorization and Accounting System. In this blog, we will see how SUSI.AI authenticates its client. Let’s first see what each term of AAA means

  • Authentication : Authentication means identifying individual user with some unique information. Susi uses email addresses for non-anonymous identity and  anonymous identity users  are identified by their host name.
  • Authorization : It refers to identifying the access rights of the user and granting permissions depending on the user’s authorization level. In Susi we have BaseUserRole as   
    • ANONYMOUS  users who have not logged in
    • USER                  logged in Users
    • PRIVILEGED       users with special rights, like. moderators
    • ADMIN              maximum right, that user is allowed to do everything . Depending on the useroles, permissions are specified.
  • Accounting : Accounting is referred as keeping track of user’s activity. Susi Server uses DAO in which accounting object is stored as JSONTray. Susi also remembers the chat log of a user.

Now that we have basic idea about AAA, let’s check how Susi authenticates its user.

public class ClientIdentity extends Client {
    
    public enum Type {
        email(true), // non-anonymous identity
        host(false); // anonymous identity users which do not authentify; they are identified by their host name
        private final boolean persistent;
      }

}

Susi has ClientIdentity class which extends to base class Client, which has a string sufficient to identify an user. The user are represented with Objects of this class. The client identification string is defined as <typeName: untypedId> pair where <typeName> denotes an authentication method and <untypedId> a name within that authentication domain.

public class Authentication {

    private JsonTray parent;
    private JSONObject json;
    private ClientCredential credential;
...
}

This credential is used as key in DAO.authentication. Parent is used for the storage object , it is null if there is no parent file (no persistency). The DAO uses credential key and implements methods like getAuthentication, getAuthorization,getAccounting taking Non null parameter ClientIdentity and returns the object of respective classes. The method setExpireTime sets an expire time for anonymous users and tokens after end of duration in time seconds passed as parameter the Authentication expires.

public class DAO {
// AAA Schema for server usage
    private static JsonTray authentication;
    private static JsonTray authorization;
    private static JsonTray accounting;
    public  static UserRoles userRoles;
..
}

The JsonTray is class to hold the volume as <String,JsonObject> pairs as a Json file. The UserRequests  class holds all the user activities. The ClientIdentity class extend the base class Client and provides an Identification String for authentication of users.

public abstract class AbstractAPIHandler extends HttpServlet implements APIHandler {
    @Override
    public abstract BaseUserRole getMinimalBaseUserRole();

}

The AbstractAPIHandler checks the permissions of the user using the userroles of and comparing it with the value minimum base role of each servlet. Thus to specify the user permission for a servlet one just need to extend servlet to AbstractAPIHandler and  Override the getMinimalBaseUserRole method.

 public static ClientIdentity getIdentity(HttpServletRequest request, HttpServletResponse response, Query query) {

if(authentication.getIdentity() != null && authentication.checkExpireTime())   // check if login cookie is set
return authentication.getIdentity();
else if(request.getSession().getAttribute("identity") != null){ // check session is set
return (ClientIdentity) request.getSession().getAttribute("identity");
else if (request.getParameter("access_token") != null){ // check if access_token is valid
 return authentication.getIdentity();
else
 return getAnonymousIdentity(query.getClientHost());
}

It also implements method getIdentity()  which checks a request for valid login data, an existing session, a cookie or an access token and returns user identity if some login is active, otherwise the anonymous identity.  

This is how Susi uses credential to authenticate users and use it for accounting and authorization. The endpoints provided by server are used by Android and web clients. Susi accounts service is at  http://accounts.susi.ai. For more details do visit code repository and join gitter chat channel for discussions.

Resources

Adding API endpoint to SUSI.AI for Skill Historization

SUSI Skill CMS is an editor to write and edit skill easily. It follows an API centric approach where the Susi server acts as API server. Using Skill CMS we can browse history of a skill, where we get commit ID, commit message  and name the author who made the changes to that skills. In this blogpost we will see how to fetch complete commit history of a skill in the susi skill repository. A skill is a set of intents. One text file represents one skill, it may contain several intents which all belong together. Susi skills are stored in susi_skill_data repository. We can access any skill based on four tuples parameters model, group, language, skill.  For managing version control in skill data repository, the following dependency is added to build.gradle . JGit is a library which implements the Git functionality in Java.

dependencies {
 compile 'org.eclipse.jgit:org.eclipse.jgit:4.6.1.201703071140-r'
}

To implement our servlet we need to extend our servlet to AbstractAPIHandler. In Susi Server, an abstract class AbstractAPIHandler extending HttpServelets and implementing API handler interface is provided.

public class HistorySkillService extends AbstractAPIHandler implements APIHandler {}

The AbstractAPIHandler checks the permissions of the user using the userroles of and comparing it with the value minimum base role of each servlet. Thus to specify the user permission for a servlet we need Override the getMinimalBaseUserRole method.

 @Override
    public BaseUserRole getMinimalBaseUserRole() {
        return BaseUserRole.ANONYMOUS;
    }

UserRoles can be Admin, Privilege, User, Anonymous. In our case it is Anonymous. A User need not to log in to access this endpoint.

  @Override
    public String getAPIPath() {
        return "/cms/getSkillHistory.json";
    }

This methods sets the api endpoint path. One need to send requests at http://api.susi.ai/cms/getSkillHistory.json to get the modification history of skill. Next we will implement The ServiceImpl method where we will be processing the user request and giving back the service response.

@Override
    public ServiceResponse serviceImpl(Query call, HttpServletResponse response, Authorization rights, final JsonObjectWithDefault permissions) {

        String model_name = call.get("model", "general");
        File model = new File(DAO.model_watch_dir, model_name);
        String group_name = call.get("group", "knowledge");
        File group = new File(model, group_name);
        String language_name = call.get("language", "en");
        File language = new File(group, language_name);
        String skill_name = call.get("skill", "wikipedia");
        File skill = new File(language, skill_name + ".txt");
        JSONArray commitsArray;
        commitsArray = new JSONArray();
        String path = skill.getPath().replace(DAO.model_watch_dir.toString(), "models");
        //Add to git
        FileRepositoryBuilder builder = new FileRepositoryBuilder();
        Repository repository = null;
        try {
            repository = builder.setGitDir((DAO.susi_skill_repo))
                    .readEnvironment() // scan environment GIT_* variables
                    .findGitDir() // scan up the file system tree
                    .build();
            try (Git git = new Git(repository)) {
                Iterable<RevCommit> logs;
                logs = git.log().addPath(path).call();
                int i = 0;
                for (RevCommit rev : logs) {
                    commit = new JSONObject();
                    commit.put("commitRev", rev);
                    commit.put("commitName", rev.getName());
                    commit.put("commitID", rev.getId().getName());
                    commit.put("commit_message", rev.getShortMessage());
                    commit.put("author",rev.getAuthorIdent().getName());
                    commitsArray.put(i, commit);
                    i++;
                } success=true;
            } catch (GitAPIException e) {
                e.printStackTrace();
                success=false;
           } if(commitsArray.length()==0){
            success=false;
        }
        JSONObject result = new JSONObject();
        result.put("commits",commitsArray);
        result.put("success",success);
        return new ServiceResponse(result);
    }

To access any skill we need parameters model, group, language. We get this through call.get method where first parameter is the key for which we want to get the value and second parameter is the default value. Based on received model, group and language browse files in that folder we build the susi_skill_data repository path read the git variables and scan up the file system tree using FileRepositoryBuilder build() method. Next we fetch all the logs of the skill file and store them in json commits array and finally pass as a server response with success messages. In case of exceptions, pass service with success flags as false.

We have successfully implemented the servlet. Check the working of endpoint by sending request like http://api.susi.ai/cms/getSkillHistory.json?model=general&group=knowledge&language=en&skill=bitcoin and checking the response.

Susi skill cms uses this endpoint to fetch the skill history, try it out at http://skills.susi.ai/browseHistory

Resources

Using SUSI AI Server to Store User Feedback for a Skill

User feedback is valuable information that plays an important  role in improving the quality of service. In SUSI AI server we are planning to make a feedback mechanism to see if the user liked the answer or not. The result of that user input (which can be given using a vote button) will be then learned to enhance the future use of the rule. So as a first step for implementation of  skill rating system with guided learning, we need to store the user rating of a skill . In this blogpost we will learn how to make an endpoint for getting skill rating from user. This API endpoint will be used  by its web and mobile clients.
Before the implementation of API  let’s look how data is stored in SUSI AI Susi_server uses DAO in which skill rating is stored as JSONTray. 

 public JsonTray(File file_persistent, File file_volatile, int cachesize) throws IOException {
        this.per = new JsonFile(file_persistent);
        this.vol = new CacheMap<String, JSONObject>(cachesize);
        this.file_volatile = file_volatile;
        if (file_volatile != null && file_volatile.exists()) try {
            JSONObject j = JsonFile.readJson(file_volatile);
            for (String key: j.keySet()) this.vol.put(key, j.getJSONObject(key));
        } catch (IOException e) {
            e.printStackTrace();
        }
    }

JsonTray takes three parameters the persistent file, volatile file and cache size to store them as cache map in String and JsonObject pairs. The HttpServlet class which provides methods, such as doGet and doPost, for handling HTTP-specific services.In Susi Server an abstract class AbstractAPIHandler extending HttpServelets and implementing API handler interface is provided. Next we will inherit our RateSkillService class from AbstractAPIHandler and implement APIhandler interface.

public class RateSkillService extends AbstractAPIHandler implements APIHandler {
    private static final long serialVersionUID =7947060716231250102L;
    @Override
    public BaseUserRole getMinimalBaseUserRole() {
        return BaseUserRole.ANONYMOUS;
    }

    @Override
    public JSONObject getDefaultPermissions(BaseUserRole baseUserRole) {
        return null;
    }

    @Override
    public String getAPIPath() {
        return "/cms/rateSkill.json";
    }

}

The getMinimalBaseRole method tells the minimum Userrole required to access this servlet it can also be ADMIN, USER. In our case it is Anonymous. A User need not to log in to access this endpoint. The getAPIPath() methods sets the API endpoint path, it gets appended to base path which is 127.0.0.1:4000/cms/rateSkill.json for local host .

Next we will implement serviceImpl method

  @Override
    public ServiceResponse serviceImpl(Query call, HttpServletResponse response, Authorization rights, final JsonObjectWithDefault permissions) {

        String model_name = call.get("model", "general");
        File model = new File(DAO.model_watch_dir, model_name);
        String group_name = call.get("group", "knowledge");
        File group = new File(model, group_name);
        String language_name = call.get("language", "en");
        File language = new File(group, language_name);
        String skill_name = call.get("skill", null);
        File skill = new File(language, skill_name + ".txt");
        String skill_rate = call.get("rating", null);

        JSONObject result = new JSONObject();
        result.put("accepted", false);
        if (!skill.exists()) {
            result.put("message", "skill does not exist");
            return new ServiceResponse(result);

        }
        JsonTray skillRating = DAO.skillRating;
        JSONObject modelName = new JSONObject();
        JSONObject groupName = new JSONObject();
        JSONObject languageName = new JSONObject();
        if (skillRating.has(model_name)) {
            modelName = skillRating.getJSONObject(model_name);
            if (modelName.has(group_name)) {
                groupName = modelName.getJSONObject(group_name);
                if (groupName.has(language_name)) {
                    languageName = groupName.getJSONObject(language_name);
                    if (languageName.has(skill_name)) {
                        JSONObject skillName = languageName.getJSONObject(skill_name);
                        skillName.put(skill_rate, skillName.getInt(skill_rate) + 1 + "");
                        languageName.put(skill_name, skillName);
                        groupName.put(language_name, languageName);
                        modelName.put(group_name, groupName);
                        skillRating.put(model_name, modelName, true);
                        result.put("accepted", true);
                        result.put("message", "Skill ratings updated");
                        return new ServiceResponse(result);
                    }
                }
            }
        }
        languageName.put(skill_name, createRatingObject(skill_rate));
        groupName.put(language_name, languageName);
        modelName.put(group_name, groupName);
        skillRating.put(model_name, modelName, true);
        result.put("accepted", true);
        result.put("message", "Skill ratings added");
        return new ServiceResponse(result);

    }

    /* Utility function*/
    public JSONObject createRatingObject(String skill_rate) {
        JSONObject skillName = new JSONObject();
        skillName.put("positive", "0");
        skillName.put("negative", "0");
        skillName.put(skill_rate, skillName.getInt(skill_rate) + 1 + "");
        return skillName;
    }

 

One can access any skill based on four tuples parameters model, group, language, skill. Before rating a skill we must ensure whether it exists or not. We can get the required parameters through call.get() method where first parameter is the key for which we want to get the value and second parameter is the default value. If skill.exists() method return false we generate error message stating “No such skill exists”. Otherwise check if the skill exist in our skillRating.json file if so, update the current ratings otherwise create a new json object and add it to rating file based on model, group and language. After successful implementation go ahead and test your endpoint on http://localhost:4000/cms/rateSkill.json?model=general&group=knowledge&skill=who&rating=positive

You can also check for the updated json file in  susi_server/data/skill_rating/skillRating.json 

{"general": {
 "assistants": {"en": {
   "language_translation": {
     "negative": "1",
     "positive": "0"
  }}},
 "smalltalk": {"en": {
   "aboutsusi": {
   "negative": "0",
   "positive": "1"
 }}},
 "knowledge": {"en": {
   "who": {
     "negative": "2",
     "positive": "4"
   }}}
}}

And if the skill is not present if will generate error message

We have successfully implemented the API endpoint for storing the user skill’s feedback. For more information take a look at Susi server and join gitter chat channel for discussions.

Resources 

Auto deployment of SUSI Skill CMS on gh pages

Susi Skill CMS is a web application framework to edit susi skills. It is currently in development stage, hosted on http://skills.susi.ai. It is built using ReactJS . In this blogpost we will see how to automatically deploy the repository on gh pages.
Setting up the project
Fork susi_skill_cms repository and clone it to your desktop, make sure you have node and npm versions greater than 6 and 3 respectively. Next go to cloned folder and install all the dependencies by running :

:$ npm install

Next run on http://localhost:3000 by running the command

:$ npm run start

To auto deploy changes on gh-pages branch, we need to setup Travis for the project. Register yourself on https://travis-ci.org/ and turn on the Travis for this repository. Next add .travis.yml in the root directory of the source folder.  

sudo: required
dist: trusty
language: node_js
node_js:
  - 6

before_install:
  - export CHROME_BIN=chromium-browser
  - export DISPLAY=:99.0
  - sh -e /etc/init.d/xvfb start

before_script:
  - npm run build

script:
  - npm run test

after_success:
  - bash ./deploy.sh

cache:
  directories: node_modules

# safelist
branches:
  only:
  - master 

Source: https://github.com/fossasia/susi_skill_cms/blob/master/.travis.yml

The travis configuration files will ensure that the project is building for every changes made, using npm run test command, in our case it will only consider changes made on master branch , if you want to watch other branches to add the respective branch name in travis configurations. After checking for build passing we need to automatically push the changes made for which we will use a bash script.

#!/bin/bash

SOURCE_BRANCH="master"
TARGET_BRANCH="gh-pages"

# Pull requests and commits to other branches shouldn't try to deploy.
if [ "$TRAVIS_PULL_REQUEST" != "false" -o "$TRAVIS_BRANCH" != "$SOURCE_BRANCH" ]; then
    echo "Skipping deploy; The request or commit is not on master"
    exit 0
fi

# Save some useful information
REPO=`git config remote.origin.url`
SSH_REPO=${REPO/https:\/\/github.com\[email protected]:}
SHA=`git rev-parse --verify HEAD`

ENCRYPTED_KEY_VAR="encrypted_${ENCRYPTION_LABEL}_key"
ENCRYPTED_IV_VAR="encrypted_${ENCRYPTION_LABEL}_iv"
ENCRYPTED_KEY=${!ENCRYPTED_KEY_VAR}
ENCRYPTED_IV=${!ENCRYPTED_IV_VAR}
openssl aes-256-cbc -K $encrypted_2662bc12c918_key -iv $encrypted_2662bc12c918_iv -in deploy_key.enc -out ../deploy_key -d
chmod 600 ../deploy_key
eval `ssh-agent -s`
ssh-add ../deploy_key

# Cloning the repository to repo/ directory,
# Creating gh-pages branch if it doesn't exists else moving to that branch
git clone $REPO repo
cd repo
git checkout $TARGET_BRANCH || git checkout --orphan $TARGET_BRANCH
cd ..

# Setting up the username and email.
git config user.name "Travis CI"
git config user.email "$COMMIT_AUTHOR_EMAIL"

# Cleaning up the old repo's gh-pages branch except CNAME file and 404.html
find repo/* ! -name "CNAME" ! -name "404.html" -maxdepth 1  -exec rm -rf {} \; 2> /dev/null
cd repo

git add --all
git commit -m "Travis CI Clean Deploy : ${SHA}"

git checkout $SOURCE_BRANCH

# Actual building and setup of current push or PR.
npm install
npm run build
mv build ../build/

git checkout $TARGET_BRANCH
rm -rf node_modules/
mv ../build/* .
cp index.html 404.html

# Staging the new build for commit; and then committing the latest build
git add -A
git commit --amend --no-edit --allow-empty

# Deploying only if the build has changed
if [ -z `git diff --name-only HEAD HEAD~1` ]; then

  echo "No Changes in the Build; exiting"
  exit 0

else
  # There are changes in the Build; push the changes to gh-pages
  echo "There are changes in the Build; pushing the changes to gh-pages"

  # Actual push to gh-pages branch via Travis
  git push --force $SSH_REPO $TARGET_BRANCH
fi

Source : Bash script for automatic deployment

This bash script will enable travis ci user to push changes to gh pages, for this we need to store the credentials of the repository in encrypted form. To get the public/ private rsa keys use the following command

ssh-keygen -t rsa -b 4096 -C "[email protected]"

It will generate keys in .ssh/id_rsa folder in your home repository.

Make sure you do not enter any passphrase while generating credentials otherwise travis will get stuck at time of decrypting the keys. Copy the public key and deploy the key to repository by visiting  https://github.com/<your name>/<your repo>/settings/keys


Next install travis for encryption of keys.

sudo apt install ruby ruby-dev
sudo gem install travis

Encrypt your private deploy_key and add it to root of your repository using command

travis encrypt-file deploy_key

After successful encryption, you will see a message

Please add the following to your build script (before_install stage in your .travis.yml, for instance):

openssl aes-256-cbc -K $encrypted_2662bc12c918_key -iv $encrypted_2662bc12c918_iv -in deploy_key.enc -out ../deploy_key -d

Add the above generated script in travis and push the changes on your master branch. Do not push the deploy_key only the encryption file deploy_key.enc
Finally, add the deploy link of gh pages in package.json of your using key “homepage”.

 "homepage": "http://skills.susi.ai/"

And in scripts of package.json add

"deploy": "gh-pages -d build",

Commit and push your changes and from now onward all your changes will be automatically pushed to gh pages branch. For contribution visit Susi_Skill_CMS.

Resources

How SUSI Analyzes A Given Response

Ever wondered where SUSI’s answers come from? Now Susi has ability to do an answer analysis. To get that analysis, just ask susi “analysis”. This will set susi into an analysis mode, will tell where the latest answer came from and will give you the link for improving the skill.

Let’s check out how Susi analysis work. The skill for analysis is defined  en_0001_foundation.txt  as following

analysis|analyse|analyze|* analysis|* analyse|* analyze|analysis *|analyse *|analyze *
My previous answer is defined in the skill $skill$. You can help to improve this skill and <a href="$skill_link$" target="_blank"> edit it in the code repository here.</a>

$skill$ and $skill_link$ are the variable compiled using

public static final Pattern variable_pattern = Pattern.compile("\\$.*?\\$");

These variables are memorized in Susi cognition. A cognition is the combination of a query of a user with the response of susi.

SusiThought dispute = new SusiThought();
List<String> skills = clonedThought.getSkills();
 if (skills.size() > 0) {
    dispute.addObservation("skill", skills.get(0));
    dispute.addObservation("skill_link",getSkillLink(skills.get(0)));
   }

Susi Thought is a piece of data that can be remembered. The structure of the thought is modeled as a table in which information contained in it is organized in rows and columns.

 public SusiThought addObservation(String featureName, String observation) ;

One can memorize using addObservation() method.  It takes two parameter featureName the object key and observation the object value. It is a table of information pieces as a set of rows which all have the same column names. It inserts the new data always in front of existing similar data rather than overwriting them.

 public String getSkillLink(String skillPath) {
       String link=skillPath;
        if(skillPath.startsWith("/susi_server")) {
            link ="https://github.com/fossasia/susi_server/blob/development" + skillPath.substring("/susi_server".length());
        } else if (skillPath.startsWith("/susi_skill_data")) {
            link = "https://github.com/fossasia/susi_skill_data/blob/master" + skillPath.substring("/susi_skill_data".length());
        }
        return link;
    }

The getSkillLink is a utitlity method to return the link of the skill source github repository based on skillPath.

private String skill;
SusiThought recall;
final SusiArgument flow = new SusiArgument().think(recall);
this.skill = origin.getAbsolutePath();
 if (this.skill != null && this.skill.length() > 0) flow.addSkill(this.skill);

The source of the skill gets added in SusiIntent.java using getAbsolutePath() method which resolves the skill path in the filesystem. Intent  considers the key from the user query, matches the intent tokens to get the optimum result and produces json like

 "data": [
      {
        "object": "If you spend too much time thinking about a thing, you'll never get it done.",
        "0": "tell me a quote",
        "token_original": "quote",
        "token_canonical": "quote",
        "token_categorized": "quote",
        "timezoneOffset": "-330",
        "answer": "When you discover your mission, you will feel its demand. It will fill you with enthusiasm and a burning desire to get to work on it. ",
        "skill_link": "https://github.com/fossasia/susi_skill_data/blob/master/models/general/entertainment/en/quotes.txt",
        "query": "tell me a quote",
        "skill": "/susi_skill_data/models/general/entertainment/en/quotes.txt"
      },

The getskills() method returns list of skill from json which are later added for memorization.

    public List<String> getSkills() {
        List<String> skills = new ArrayList<>();
        getSkillsJSON().forEach(skill -> skills.add((String) skill));
        return skills;
    }

This is how Susi is able to fetch  where the answer came from. Next time when you have a chat with susi do check skill analysis and add your ideas to improve the skill. Take a look at Susi_skill_data for more skills and  read this tutorial  for creating skills for susi.

Resources

Using SUSI AI Accounting Object to Write User Settings

SUSI Server uses DAO in which accounting object is stored as JSONTray. SUSI clients are using this accounting object for user settings data. In this blogpost we will focus on how to use accounting JSONTray to write the user settings, so that a client can use such endpoint to store the user related settings in Susi server. The Susi server provides the required API endpoints to its web and mobile clients. Before starting with the implementation of servlet let’s take a look at Accounting.java file, to check how Susi server stores the accounting data.

public class Accounting {

        private JsonTray parent;
        private JSONObject json;
        private UserRequests requests;
        private ClientIdentity identity;
    ...
}

 

The JsonTray is class to hold the volume as <String,JsonObject> pairs as a Json file. The UserRequests  class holds all the user activities. The ClientIdentity class extend the base class Client and provides an Identification String for authentication of users. Now that we have understood about accounting in SUSI server let’s proceed for making an API endpoint to Store Webclient User settings. To make an endpoint we will use the HttpServlet class which provides methods, such as doGet and doPost, for handling HTTP-specific services. We will inherit our ChangeUserSettings class from AbstractAPIHandler yand implement APIhandler interface. In Susi Server the AbsrtactAPI handler extends a HTTPServlet which implements doGet and doPost ,all servlet in SUSI Server extends this class to increase code reusability.  

Since a User has to store its setting, set the minimum base role to access this endpoint to User. Apart from ‘User’ there are Admin and Anonymous roles too.

   @Override
    public BaseUserRole getMinimalBaseUserRole() {
        return BaseUserRole.USER;
    }

Next set the path for using this endpoint, by overriding getAPIPath method().

 @Override
    public String getAPIPath() {
        return "/aaa/changeUserSettings.json";
    }

We won’t be dealing with getdefault permissions so null can be return.

  @Override
    public JSONObject getDefaultPermissions(BaseUserRole baseUserRole) {
        return null;
    }

Next we implement serviceImpl method which takes four parameters the query, response, authorization and default permissions.

@Override
    public ServiceResponse serviceImpl(Query query, HttpServletResponse response, Authorization authorization, JsonObjectWithDefault permissions) throws APIException {
       String key = query.get("key", null);
       String value =query.get("value", null);
       if (key == null || value == null ) {
           throw new APIException(400, "Bad Service call, key or value parameters not provided");
       } else {
           if (authorization.getIdentity() == null) {
               throw new APIException(400, "Specified User Setting not found, ensure you are logged in");
           } else {
               Accounting accounting = DAO.getAccounting(authorization.getIdentity());
               JSONObject jsonObject = new JSONObject();
               jsonObject.put(key, value);
               if (accounting.getJSON().has("settings")) {
                   accounting.getJSON().getJSONObject("settings").put(key, value);
               } else {
                   accounting.getJSON().put("settings", jsonObject);
               }
               JSONObject result = new JSONObject();
               result.put("message", "You successfully changed settings to your account!");
               return new ServiceResponse(result);
           }
       }

    }

We will be storing the setting in Json object using key, value pairs. Take the values from user using query.get(“param”,”default value”) and set the default value to null. So that in case the parameters are not present the servlet can return “Bad service call”. To get the accounting object user identity string given by authorization.getIdentity() method is used. Now check if the same user settings is already present, if yes, overwrite it and if not append a new Json object with received key and value. And return the success message through ServiceResponse method.

Proceed to test the working of endpoint at http://127.0.0.1:4000/aaa/changeUserSettings.json?key=theme&value=dark and see if it’s stored using   http://127.0.0.1:4000/aaa/listUserSettings.json.

You have successfully created an endpoint to store user settings and  enhanced Susi Server, take a look and contribute to Susi Server.

Resources

Adding a new Servlet/API to SUSI Server for Skill Wiki

Susi skill wiki is an editor to write and edit skill easily. It follows an API-centric approach where the Susi server acts as API server and a web front-end  act as the client for the API and provides the user interface. A skill is a set of intents. One text file represents one skill, it may contain several intents which all belong together.

The schema for storing a skill is as following:

Using this, one can access any skill based on four tuples parameters model, group, language, skill.  To achieve this on server side let’s create an API endpoint to list all skills based on given model, groups and languages. To check the source for this endpoint clone the susi_server repository from here.

git clone https://github.com/fossasia/susi_server.git

Have a look at documentation for more information about Susi Server.

The Servlet java file is placed in susi_server/ai/susi/server/api/cms/ListSkillService. To implement the endpoint we will use the HttpServlet class which provides methods, such as doGet and doPost, for handling HTTP-specific services.

In Susi Server an abstract class AbstractAPIHandler extending HttpServelets and implementing API handler interface is provided.  Next we will inherit our ListSkillService class from AbstractAPIHandler and implement APIhandler interface.

To implement our servlet we will be overriding 4 methods namely

  • Minimal Base User role 

public BaseUserRole getMinimalBaseUserRole() {
return BaseUserRole.ANONYMOUS;
}

This method tells the minimum Userrole required to access this servlet it can also be ADMIN, USER. In our case it is Anonymous. A User need not to log in to access this endpoint.

  • Default Permissions  

public JSONObject getDefaultPermissions(BaseUserRole baseUserRole) {
return null;
}

This method returns the default permission attached with base user role, our servlets has nothing to do with it, therefore we can simply return null for this case.

  • The API Path 

public String getAPIPath() {
return "/cms/getSkillList.json";
}

This methods sets the API endpoint path, it gets appended to base path which is 127.0.0.1:4000/cms/getSkillList.json for the local host and http://api.susi.ai/cms/getSkillList.json for the server.

  • The ServiceImpl method 

public ServiceResponse serviceImpl(Query call, HttpServletResponse response, Authorization rights, final JsonObjectWithDefault permissions) {

        String model_name = call.get("model", "general");
        File model = new File(DAO.model_watch_dir, model_name);
        String group_name = call.get("group", "knowledge");
        File group = new File(model, group_name);
        String language_name = call.get("language", "en");
        File language = new File(group, language_name);

        ArrayList fileList = new ArrayList();
        fileList =  listFilesForFolder(language, fileList);
        JSONArray jsArray = new JSONArray(fileList);

        JSONObject json = new JSONObject(true)
                .put("model", model_name)
                .put("group", group_name)
                .put("language", language_name)
                .put("skills", jsArray);
        return new ServiceResponse(json);

    }

    ArrayList listFilesForFolder(final File folder,  ArrayList fileList) {

        File[] filesInFolder = folder.listFiles();
        if (filesInFolder != null) {
            for (final File fileEntry : filesInFolder) {
                if (!fileEntry.isDirectory()) {
                    fileList.add(fileEntry.getName()+"");
                }
            }
        }
        return  fileList;
    }

To access any skill we need parameters model, group, language. We get this through call.get method where first parameter is the key for which we want to get the value and second parameter is the default value. Based on received model, group and language browse files in that folder and put them in Json array to return the Service Json response.
That’s all you have successfully implemented the API endpoint to list all the skills in given model group and language. Move on and test it.

And see the resultsTo access any skill we need  parameters model, group, language. We get this through call.get method where first parameter is the key for which we want to get the value and second parameter is the default value. Based on received model, group and language browse files in that folder and put them in Json array to return the Service Json response.

 You have  successfully implemented the API endpoint to list all the skills in given model group and language. Move on and test it.

It’s easy isn’t it you have learnt how to add an endpoint in the server, it’s time to go ahead and create more endpoints to enhance Susi Server, take a look and contribute to Susi Server.