Susi

This is a concept for the management of the skill repository aka The “SUSI Skill CMS“.

With SUSI we are building a personal assistant where the users are able to write and edit skills in the easiest way that we can think of. To do that we have to develop a very simple skill language and a very simple skill editor

The skill editor should be done as a ‘wiki’-like content management system (cms). To create the wiki, we follow an API-centric approach. The SUSI server acts as an API server with a web front-end which acts as a client of the API and provides the user interface.

The skill editor will be ‘expert-centric’, an expert is a set of skills. That means if we edit one text file, that text file represents one expert, it may contain several skills which all belong together.

An ‘expert’ is stored within the following ontology:

model  >  group  >  language  >  expert  >  skill

To Implement the CMS wiki system we need versioning with a working AAA System. To implement versioning we used JGit. JGit is an EDL licensed, lightweight, pure Java library implementing the Git version control system.

So I included a Gradle dependency to add JGit to the SUSI Server project.

compile 'org.eclipse.jgit:org.eclipse.jgit:4.6.1.201703071140-r'

Now the task was to execute git commands when the authorised user makes changes in any of the expert. The possible changes in an expert can be

1. Creating an Expert
2. Modifying an existing Expert
3. Deleting an Expert

1. git add <filename>

2. git commit -m “commit message”

Real Example in SUSI Server

This is the code that every servlet shares. It defines the base user role set a URL endpoint to trigger the endpoint

public class ModifyExpertService extends AbstractAPIHandler implements APIHandler {
    @Override
    public String getAPIPath() {
        return "/cms/modifyExpert.json";
    }

This is the part where we do all the processing of the URL parameters and store their versions. This method takes the “Query call” and then extracts the “get” parameters from it.
For the functioning of this service, we need 5 things, “model”, “group”, “language”, “expert” and the “commit message”.

@Override
public ServiceResponse serviceImpl(Query call, HttpServletResponse response, Authorization rights, final JsonObjectWithDefault permissions) {

    String model_name = call.get("model", "general");
    File model = new File(DAO.model_watch_dir, model_name);
    String group_name = call.get("group", "knowledge");
    File group = new File(model, group_name);
    String language_name = call.get("language", "en");
    File language = new File(group, language_name);
    String expert_name = call.get("expert", null);
    File expert = new File(language, expert_name + ".txt");

Then we need to open your SUSI Skill DATA repository and commit the new file in it. Here we call the functions of JGit, which do the work of git add and git commit.

FileRepositoryBuilder builder = new FileRepositoryBuilder();
Repository repository = null;
try {

    repository = builder.setGitDir((DAO.susi_skill_repo))
            .readEnvironment() // scan environment GIT_* variables
            .findGitDir() // scan up the file system tree
            .build();

    try (Git git = new Git(repository)) {

The code above opens our local git repository and creates an object “git”. Then we perform further operations on “git” object. Now we add our changes to “git”. This is similar to when we run “git add . ”

    git.add()
                .addFilepattern(expert_name)
                .call();

Finally, we commit the changes. This is similar to “git commit -m “message”.

    git.commit()
                .setMessage(commit_message)
                .call();

At last, we return the success object and set the “accepted” value as “true”.

    json.put("accepted", true);
        return new ServiceResponse(json);
    } catch (GitAPIException e) {
        e.printStackTrace();
    }

Resources

JGit documentation : https://eclipse.org/jgit/documentation/

SUSI Server : https://github.com/fossasia/susi_server

The core of SUSI Clients is the SUSI server that holds the “intelligence” and “personality” of SUSI AI. SUSI Server is in JAVA and it uses the concepts of Servlets heavily. While implementing server for SUSI, we have used JAVA as the backend language and hence are using servlets and Servlet containers heavily.

The problem that servlets are solving is regarding how to generate dynamic content on every request if the request comes with different parameters.  This is not possible by simply using HTML.

Servlets are widely used for dynamic content and have inbuilt support for HTTP (Hypertext Transfer Protocol).

In this blog post, along with basics of servlets and servlet containers, I am explaininghow to make classes for custom servlets. I am using an example of AbstractAPIHandler class that we have used in SUSI server for clear picture.

Web Server

Before we can understand a  servlet container, we need to understand what is a web server.

A web server uses HTTP protocol to transfer data. When a user types in a URL in a browser or any client, he first sees loading and then see the content of the page. So actually there is a server that is sending the content of the page to the user. The transfer of data is in HTTP protocol.

Servlet Container

From the example above, a user can request static pages from the server. Hence the pages will be very basic and we cannot do much with user inputs and dynamic data such as making a real time chat bot. So the idea of having Servlet container is using Java to dynamically generate the content of a web page from the server side.  

The flow of Control

The servlet container manages servlets through servlet life cycles. The container calls a servlet as soon as it receives an HTTP request. Then the servlet does the processing of the data which it has received from the request. the container sends the HTTP response back to the client and then the clients render it to show the final output.

The flowchart below explains how a request is processed from a browser to a servlet container and then to a servlet. It shows the usage of a generic Servlet.

Similarly in SUSI-Server we have a “AbstractAPIHandler” which is a generic servlet that every other servlet inherits.

Structure of a Servlet

public class ServletName extends HttpServlet {

public void init() throws ServletException {
// Servlet Initialization
}

protected void service(HttpServletRequest req,
HttpServletResponse resp)
throws ServletException, IOException {
// Code for the Service Method
}
/**
* Process a GET request
*/
protected void doGet(HttpServletRequest request,
HttpServletResponse response)
throws IOException, ServletException {
// Code for the doGet() method
}
/**
* Process a POST request
*/
protected void doPost(HttpServletRequest request,
HttpServletResponse response)
throws IOException, ServletException {
// Code for the doPost() method

}
/**
* Process a PUT request
*/
protected void doPut(HttpServletRequest req,
HttpServletResponse resp)
throws ServletException, IOException {
//Code for the doPut() method

}
}

Abstract API Handler

public abstract class AbstractAPIHandler extends HttpServlet implements APIHandler {
}

The function below sets the Minimum Base Role of the SUSI user. We can set which servlet is accessible by which user.

@Override
public abstract BaseUserRole getMinimalBaseUserRole();

The function below gets the Default Permissions of a user Role in SUSI Server. The permissions for each SUSI user can be different.

@Override
public abstract JSONObject getDefaultPermissions(BaseUserRole baseUserRole);

The function below finally sets whatever output we want to show as a response when we query the SUSI Server.

   public abstract ServiceResponse serviceImpl(Query post,  HttpServletResponse response, Authorization rights, final JsonObjectWithDefault permissions) throws APIException;

Each Servlet in SUSI Server extends this class (AbstractAPIHandler.java). This also increases code reusability.

A Servlet is not a just a simple java class. You cannot run a servlet using javac or by running main() function. In order to run a servlet, you have to deploy this Servlet on a web server.

References

SUSI Server : https://github.com/fossasia/susi_server/

Docs: http://docs.oracle.com/javaee/6/tutorial/doc/bnafd.html