SUSI Skill Data is the storage place for susi skills. All the skills are stored in these repository follows schema Models -> Groups -> Languages -> Skill -> Intents. Using this, one can access any skill based on four tuples parameters model, group, language, skill. The SUSI skill CMS is an editor to write and edit skill easily. It follows an API-centric approach where the SUSI server acts as API server and a web front-end act as the client for the API and provides the user interface. In this blog we will see how SUSI server updates the skill data repository.
For implementing versioning of SUSI Skills, server uses JGit. JGit is an EDL licensed, lightweight, pure Java library implementing the Git version control system. To include a Gradle dependency to add JGit to the SUSI Server project.
compile 'org.eclipse.jgit:org.eclipse.jgit:4.6.1.201703071140-r'
Next we initialize the SUSI Skill Data repository in DAO.
susi_skill_repo = new File(data_dir.getParentFile().getParentFile(), "susi_skill_data/.git"); repository = builder.setGitDir((DAO.susi_skill_repo)) .readEnvironment() // scan environment GIT_* variables .findGitDir() // scan up the file system tree .build();
The code above opens our local git repository and creates an object “git”. Then we perform further operations on “git” object. Now we add our changes to “git”. This is similar to when we run any command using git. Next, to pull the latest changes we define a pull method catching the IOException,
public static void pull(Git git) throws IOException { try { PullResult pullResult = git.pull().call(); MergeResult mergeResult = pullResult.getMergeResult(); if (mergeResult!=null && mergeResult.getConflicts()!=null) { pullStatus =false; // we have conflicts send email to admin try { EmailHandler.sendEmail(getConfig("skill_repo.admin_email",""), "SUSI Skill Data Conflicts", getConflictsMailContent(mergeResult)); } catch (Throwable e) { e.printStackTrace(); } } else { PushCommand push = git.push(); push.setCredentialsProvider(new UsernamePasswordCredentialsProvider(getConfig("github.username", ""), getConfig("github.password",""))); push.call(); } } catch (GitAPIException e) { throw new IOException (e.getMessage()); }
We check if there are any merge conflicts while merging the latest changes and local commits, if there are any we send the email to administrator to resolve them manually. Otherwise we push the commits to update the local git repository. The credentials currently are stored in config.properties file, and is used using the getConfig method.
To pull Susi_SKill_Data every minute we add a thread in DAO initialization method.
log("Starting Skill Data pull thread"); Thread pullThread = new Thread() { @Override public void run() { while (DAO.pullStatus) { try { Thread.sleep(getConfig("skill_repo.pull_delay", 60000)); } catch (InterruptedException e) { break; } try { DAO.pull(DAO.getGit()); } catch (Exception e) { DAO.pullStatus =false; Log.getLog().warn("SKILL PULL THREAD", e); } } } }; pullThread.start();
The thread sleeps for a default of 60 seconds, and updates the skill_data until any exception is faced. In case of exception the threads stops and administrator is informed about the git status.
This is how server pulls skill data every minute. To add the metadata to the skill visit susi_skill_data, the storage place for susi skills. For more information and complete code take a look at Susi server and join gitter chat channel for discussions.
Resources
- JGit documentation : https://eclipse.org/jgit/documentation/
- The source code SUSI Server : https://github.com/fossasia/susi_server
- Read about servlet interface – https://www.javatpoint.com/Servlet-interface
- Read about DAO- https://en.wikipedia.org/wiki/Data_access_object