Skip to content
blog.fossasia.org
  • Home
  • Projects
    • Contribute
  • Events
    • Eventyay Platform
    • Event Sponsorships
    • Event Calendar
    • FOSSASIA Summit
    • OpenTechSummit China
    • OpenTechSummit Thailand
    • OpenTechSummit Vietnam
    • Jugaad Fest India
    • Past Events
      • FOSSASIA Summit 2022
      • FOSSASIA Summit 2021
      • FOSSASIA Summit 2020
      • FOSSASIA Summit 2019
      • FOSSASIA Summit 2018
      • FOSSASIA Summit 2017
      • FOSSASIA Summit 2016
      • FOSSASIA Summit 2015
      • FOSSASIA Summit 2014
      • FOSSASIA Summit 2012
      • FOSSASIA Summit 2011
      • FOSSASIA Summit 2010
      • GNOME.Asia 2009
      • MiniDebConf Vietnam 2010
      • Sciencehack.Asia
      • Science Hack India
  • Programs
    • Programs and Opportunities
    • Jobs Opportunities
    • Program Guidelines
    • Codeheat Contest
    • University Internship Program
    • University Student Coding Programs
    • High School Student Program
    • Advanced Developer Program
    • Become a Mentor
      • Become A University Student Mentor
      • Become A High School Student Mentor
  • Shop
  • Blog
  • About
    • Jobs
    • Membership
    • Activities
    • Background & Mission
    • Best Practices
    • Licenses
    • Team
    • Code of Conduct
  • Donate
Menu Close
  • Home
  • Projects
    • Contribute
  • Events
    • Eventyay Platform
    • Event Sponsorships
    • Event Calendar
    • FOSSASIA Summit
    • OpenTechSummit China
    • OpenTechSummit Thailand
    • OpenTechSummit Vietnam
    • Jugaad Fest India
    • Past Events
      • FOSSASIA Summit 2022
      • FOSSASIA Summit 2021
      • FOSSASIA Summit 2020
      • FOSSASIA Summit 2019
      • FOSSASIA Summit 2018
      • FOSSASIA Summit 2017
      • FOSSASIA Summit 2016
      • FOSSASIA Summit 2015
      • FOSSASIA Summit 2014
      • FOSSASIA Summit 2012
      • FOSSASIA Summit 2011
      • FOSSASIA Summit 2010
      • GNOME.Asia 2009
      • MiniDebConf Vietnam 2010
      • Sciencehack.Asia
      • Science Hack India
  • Programs
    • Programs and Opportunities
    • Jobs Opportunities
    • Program Guidelines
    • Codeheat Contest
    • University Internship Program
    • University Student Coding Programs
    • High School Student Program
    • Advanced Developer Program
    • Become a Mentor
      • Become A University Student Mentor
      • Become A High School Student Mentor
  • Shop
  • Blog
  • About
    • Jobs
    • Membership
    • Activities
    • Background & Mission
    • Best Practices
    • Licenses
    • Team
    • Code of Conduct
  • Donate

SkillMetadata

Read more about the article Adding Languages Name in SUSI Skill CMS

Adding Languages Name in SUSI Skill CMS

  • Post author:saurabhjn76
  • Post published:August 30, 2017
  • Post category:FOSSASIA
  • Post comments:0 Comments

The SUSI skill CMS is an editor to write and edit skill easily. It follows an API-centric approach where the SUSI server acts as API server and a web front-end  act as the client for the API and provides the user interface. A skill is a set of intents. One text file represents one skill, it may contain several intents which all belong together. All the skills are stored in SUSI Skill Data repository and the schema is Models -> Groups -> Languages -> Skill -> Intents.

http://api.susi.ai/cms/getAllLanguages.json

To get the list of all the languages stored in Susi Skill Data repository, we used the endpoint provided by SUSI Server.

    @Override
    public UserRole getMinimalUserRole() { return UserRole.ANONYMOUS; }

The getMinimalBaseRole method tells the minimum Userrole required to access this servlet it can also be ADMIN, USER. In our case it is Anonymous. A User need not to log in to access this endpoint.

@Override
    public ServiceResponse serviceImpl(Query call, HttpServletResponse response, Authorization rights, final JsonObjectWithDefault permissions) {
        String model_name = call.get("model", "general");
        File model = new File(DAO.model_watch_dir, model_name);
        String group_name = call.get("group", "Knowledge");
        File group = new File(model, group_name);
        JSONObject json = new JSONObject(true);
        json.put("accepted", false);
        String[] languages = group.list((current, name) -> new File(current, name).isDirectory());
        JSONArray languagesArray = new JSONArray(languages);
        json.put("languagesArray", languagesArray);
        json.put("accepted", true);
        return new ServiceResponse(json);
    }

One can access based on model and group. We can get the required parameters through call.get() method where first parameter is the key for which we want to get the value and second parameter is the default value. We make a json array of languages ISO codes and send it as a response for this endpoint.

Next, to display the actual name we used a ISO-639-1 npm package for language name translation from their respective codes.

console.log(ISO6391.getName('zh')) // 'Chinese'
console.log(ISO6391.getNativeName('zh')) // '中文'

Next we use getAlllanguages.json endpoint of Susi_server

if (languages.length === 0) {
            $.ajax({
                url: 'http://api.susi.ai/cms/getAllLanguages.json',
                jsonpCallback: 'pc',
                dataType: 'jsonp',
                jsonp: 'callback',
                crossDomain: true,
                success: function (data) {

                    data = data.languagesArray
                    this.setState({ languages: data });
                    for (let i = 0; i < data.length; i++) {
                        if (ISO6391.getNativeName(data[i])) {
                            languages.push(<MenuItem
                                value={data[i]}
                                key={data[i]}
                                primaryText={ISO6391.getNativeName(data[i])} />);
                        }
                        else {
                            languages.push(<MenuItem value={data[i]}
                                                    key={data[i]}
                                                    primaryText={'Universal'} />);
                        }
                    }
                }.bind(this)
            });

and make an Ajax network call to get the ISO codes from the above endpoints. We parses the response in data object, using ISO6391 npm package we added the native language name in the drop down for the languages. We defined a Universal language with code ‘xx’, for skills which is related to mathematical and computational problems.

This is how we added language’s native form in Susi skill CMS, To add more skills or modify the existing ones visit skills.susi.ai. For more information and complete code take a look at Susi server and Susi_Skill_cms and join gitter chat channel for discussions.

Resources

  • Read more about Servets – https://www.tutorialspoint.com/servlets/
  • The source code –https://github.com/fossasia/susi_skill_cms
  • Read about servlet interface – https://www.javatpoint.com/Servlet-interface
  • Read about npm package – https://www.npmjs.com/package/iso-639-1
Continue ReadingAdding Languages Name in SUSI Skill CMS
Read more about the article Updating Skill Data Repository through SUSI Server

Updating Skill Data Repository through SUSI Server

  • Post author:saurabhjn76
  • Post published:August 21, 2017
  • Post category:FOSSASIA/GSoC/SUSI.AI
  • Post comments:0 Comments

SUSI Skill Data is the storage place for susi skills. All the skills are stored in these repository follows schema  Models -> Groups -> Languages -> Skill -> Intents. Using this, one can access any skill based on four tuples parameters model, group, language, skill. The SUSI skill CMS is an editor to write and edit skill easily. It follows an API-centric approach where the SUSI server acts as API server and a web front-end  act as the client for the API and provides the user interface. In this blog we will see how SUSI server updates the skill data repository.

For implementing versioning of SUSI Skills, server uses JGit. JGit is an EDL licensed, lightweight, pure Java library implementing the Git version control system.  To include a Gradle dependency to add JGit to the SUSI Server project.

compile 'org.eclipse.jgit:org.eclipse.jgit:4.6.1.201703071140-r'

Next we initialize the SUSI Skill Data repository in DAO.

 susi_skill_repo = new File(data_dir.getParentFile().getParentFile(), "susi_skill_data/.git");
repository = builder.setGitDir((DAO.susi_skill_repo))
           .readEnvironment() // scan environment GIT_* variables
           .findGitDir() // scan up the file system tree
           .build();

The code above opens our local git repository and creates an object “git”. Then we perform further operations on “git” object. Now we add our changes to “git”. This is similar to when we run any command using git. Next, to pull the latest changes we define a pull method catching the IOException,

public static void pull(Git git) throws IOException {
       try {
           PullResult pullResult = git.pull().call();
           MergeResult mergeResult = pullResult.getMergeResult();

           if (mergeResult!=null && mergeResult.getConflicts()!=null) {
               pullStatus =false;
               // we have conflicts send email to admin
               try {
                   EmailHandler.sendEmail(getConfig("skill_repo.admin_email",""), "SUSI Skill Data Conflicts", getConflictsMailContent(mergeResult));
               } catch (Throwable e) {
                e.printStackTrace();
               }

           } else {
               PushCommand push = git.push();
               push.setCredentialsProvider(new UsernamePasswordCredentialsProvider(getConfig("github.username", ""), getConfig("github.password","")));
               push.call();
           }
           
       } catch (GitAPIException e) {
           throw new IOException (e.getMessage());
       }

We check if there are any merge conflicts while merging the latest changes and local commits, if there are any we send the email to administrator to resolve them manually. Otherwise we push the commits to update the local git repository. The credentials currently are stored in config.properties file, and is used using the getConfig method.
To pull Susi_SKill_Data every minute we add a thread in DAO initialization method.

 log("Starting Skill Data pull thread");
       Thread pullThread = new Thread() {
           @Override
           public void run() {
               while (DAO.pullStatus) {
                   try {
                       Thread.sleep(getConfig("skill_repo.pull_delay", 60000));
                   } catch (InterruptedException e) {
                       break;
                   }
                   try {
                       DAO.pull(DAO.getGit());
                   } catch (Exception e) {
                       DAO.pullStatus =false;
                       Log.getLog().warn("SKILL PULL THREAD", e);
                   }
               }
           }
       };
       pullThread.start();

The thread sleeps for a default of 60 seconds, and updates the skill_data until any exception is faced. In case of exception the threads stops and administrator is informed about the git status.

This is how server pulls skill data every minute. To add the metadata  to the skill visit susi_skill_data, the storage place for susi skills. For more information and complete code take a look at Susi server and join gitter chat channel for discussions.

Resources

  • JGit documentation : https://eclipse.org/jgit/documentation/
  • The source code SUSI Server : https://github.com/fossasia/susi_server
  • Read about servlet interface – https://www.javatpoint.com/Servlet-interface
  • Read about DAO- https://en.wikipedia.org/wiki/Data_access_object
Continue ReadingUpdating Skill Data Repository through SUSI Server
Read more about the article Implementing Skill Page in SUSI Skill CMS

Implementing Skill Page in SUSI Skill CMS

  • Post author:saurabhjn76
  • Post published:August 21, 2017
  • Post category:FOSSASIA/GSoC/SUSI.AI
  • Post comments:0 Comments

The SUSI skill CMS is an editor to write and edit skill easily. It follows an API-centric approach where the SUSI server acts as API server and a web front-end  act as the client for the API and provides the user interface. A skill is a set of intents. One text file represents one skill, it may contain several intents which all belong together. All the skills are stored in SUSI Skill Data repository and the schema is Models -> Groups -> Languages -> Skill -> Intents. Using this, one can access any skill based on four tuples parameters model, group, language, skill.

A skill format looks like following,

::name <Skill_name>
::author <author_name>
::author_url <author_url>
::description <description> 
::dynamic_content <Yes/No>
::developer_privacy_policy <link>
::image <image_url>
::terms_of_use <link>

#Intent
User query1 query2 quer3....
!example:<The question that should be shown in public skill displays>
!expect:<The answer expected for the above example>
Answer for the user query

 

Susi Skill class in Susi Server provides parser methods for sets of intents, given as text files.  It reads the skill line by line. The SusiMind class process this JSON and stores the metadata in a map of skill path and data, which provides the API endpoint for getting the skill the metadata .    http://api.susi.ai/cms/getSkillMetadata.json?model=general&group=Knowledge&language=en&skill=cars. It takes four parameters, model, group, language, and skill. The endpoint returns null value if the skill metadata field is not present in the skill. The following is the response for model=general, group=Knowledge, language =en and skill =cars.

"skill_metadata": {
    "model": "general",
    "group": "Knowledge",
    "language": "en",
    "developer_privacy_policy": null,
    "descriptions": "A skill to tell user about cars made by a car manufacturer",
    "image": "images/cars.png",
    "author": "Uday Theja",
    "author_url": "https://github.com/uday96",
    "skill_name": "Cars",
    "terms_of_use": null,
    "dynamic_content": true,
    "examples": [
      "Know any car models manufactured by ford",
      "Know any car models manufactured by ford in the year 2016",
      "Know any car models manufactured in 2016 by ford"
    ]
  },

Image is the path to image of the skill, stored in skill data repository and dynamic content represents if the skill’s response changes based on the user inputs.
To make the required API call we first need the value of 4 parameters. We will get these as props from browse skill route.

<Link key={el}
   to={{
        pathname: '/skillPage',
state: { url: url, element: el, name: el, modelValue: self.state.modelValue, groupValue:self.state.groupValue,        languageValue:self.state.languageValue}
                             }}>

Using these props we will append the base url with all the four parameters.

 let baseUrl = 'http://api.susi.ai/cms/getSkillMetadata.json';
 url = baseUrl + '?model=' + modelValue + '&group=' + groupValue + '&language=' + languageValue + '&skill=' + name;

Next we will make a ajax call and get the data updated.

 $.ajax({
                url: url,
                jsonpCallback: 'pc',
                dataType: 'jsonp',
                jsonp: 'callback',
                crossDomain: true,
                success: function (data) {
                    self.updateData(data.skill_metadata)
                }
            });

To display data on Skill CMS we will use Material UI components. Before rendering we will check the each state if it is null , we will display the default values for it.

{this.state.imgUrl === undefined?
                            <CircleImage name={this.state.skill_name.toUpperCase()} size="250"/>:
                            <img className="avatar-img" alt="Thumbnail" src={this.state.imgUrl}/>

If the image is not present in the skill, we display the placeholder showing the name of the skill.

Otherwise, the image stored in skill_data repository is shown.

Similarly, we will display the other components using material-ui. Next, we add an edit button for editing the skill.

To make edit button we use floating action button. We link the button to skillEditor route and pass the  URL and skillname as props.

  <Link to={{
           pathname: '/skillEditor',
           state: { url: urlCode, name:name, }
   }}>

SkillEditor components uses getSkill.json endpoint and using these props make the ajax call to the endpoint like: http://api.susi.ai/cms/getSkill.json?model=general&group=Entertainment&language=en&skill=<skill_name>.

This is how skill Page is implemented in Susi skill CMS, To add more skills or modify the existing ones visit skills.susi.ai. For more information and complete code take a look at Susi server and Susi_Skill_cms and join gitter chat channel for discussions.

Resources

  • Read more about Servlets – https://www.tutorialspoint.com/servlets/
  • The source code –https://github.com/fossasia/susi_skill_cms/tree/master/src/components/SkillPage
  • Read about servlet interface – https://www.javatpoint.com/Servlet-interface
  • Read merits of Material UI- https://thenextweb.com/dd/2015/11/10/what-are-the-real-merits-of-material-design/#.tnw_hnWuEF06
Continue ReadingImplementing Skill Page in SUSI Skill CMS
Read more about the article Adding Skill Metadata in SUSI AI

Adding Skill Metadata in SUSI AI

  • Post author:saurabhjn76
  • Post published:August 14, 2017
  • Post category:FOSSASIA
  • Post comments:0 Comments

A skill is a set of intents. One text file represents one skill, it may contain several intents which all belong together. All the skills are stored in Susi Skill Data repository and the schema is Models -> Groups -> Languages -> Skill -> Intents. Using this, one can access any skill based on four tuples parameters model, group, language, skill. Susi skill CMS is an editor to write and edit skill easily. It follows an API-centric approach where the Susi server acts as API server and a web front-end act as the client for the API and provides the user interface.

::name <Skill_name>
::author <author_name>
::author_url <author_url>
::description <description> 
::dynamic_content <Yes/No>
::developer_privacy_policy <link>
::image <image_url>
::terms_of_use <link>

#Intent
User query1|query2|quer3....
!example:<The question that should be shown in public skill displays>
!expect:<The answer expected for the above example>
Answer for the user query

To give more information of a Skill above metadata was added to the text file. Where dynamic content, represents if the skill answer changes based on user inputs. And image is the path to image of the skill, stored in skill data repository. Let’s check how this is implemented in Susi Server.

  public static JSONObject readEzDSkill(BufferedReader br) throws JSONException {
  if (line.startsWith("::")) {
        int thenpos=-1;
 if (line.startsWith("::name") && (thenpos = line.indexOf(' ')) > 0) {
      skillName = line.substring(thenpos + 1).trim();
      if(skillName.length() > 0)
      json.put("skill_name",skillName);
  }
    if (line.startsWith("::author") && (thenpos = line.indexOf(' ')) > 0) {
                   authorName = line.substring(thenpos + 1).trim();
        if(authorName.length() > 0)
                        json.put("author",authorName);
                }
 if (line.startsWith("::author_url") && (thenpos = line.indexOf(' ')) > 0) {
    authorURL = line.substring(thenpos + 1).trim();
    if(authorURL.length() > 0)
    json.put("author_url",authorURL);
  }
  lastLine = ""; example = ""; expect = ""; description = ""; image = "";
                continue readloop;
            }

Susi Skill class provides parser methods for sets of intents, given as text files. It reads the skill line by line. For differentiating skill metadata with skill code we have use (::) operator. If a line starts with ::, method knows it’s skill metadata, and process it respectively. It takes out the respective substring and puts in json to be later processed by SusiMind class.

 private final Map<String, String> skillNames;
  private final Map<String, String> authors;
  private final Map<String, String> authorsUrl;
  private final Map<String, String> devloperPrivacyPolicies; 
  private final Map<String, String> termsOfUse; 
  private final Map<String,Boolean> dynamicContent; 
 if(json.has("description"))
                    this.skillDescriptions.put(intent.getSkill(), json.getString("description"));
                // skill image
                if(json.has("image"))
                    this.skillImage.put(intent.getSkill(), json.getString("image"));
                // adding skill meta data
                if(json.has("skill_name"))
                    this.skillNames.put(intent.getSkill(), json.getString("skill_name"));
                if(json.has("author"))
                    this.authors.put(intent.getSkill(), json.getString("author"));

The SusiMind class process this json and stores the metadata in a map of skill path and data.This map is used by GetSkillMetadataService class to to list the skill metadata for all the skills given its model, group and language and skill. For adding the servlet we need to inherit the service class from AbstractAPIHandler and implement APIhandler interface.In Susi Server an abstract class AbstractAPIHandler extending HttpServelets and implementing API handler interface is provided.

 @Override
    public String getAPIPath() {
        return "/cms/getSkillMetadata.json";
    }

The getAPIPath() methods sets the API endpoint path, it gets appended to base path which is 127.0.0.1:4000/cms/getSkillMetadata.json for local host. The getMinimalBaseRole method tells the minimum Userrole required to access this servlet it can also be ADMIN, USER. In our case it is Anonymous. A User need not to log in to access this endpoint.
In Serviceimpl method,

  for (Map.Entry<String, String> entry : DAO.susi.getSkillDescriptions().entrySet()) {
            String path = entry.getKey();
            if ((path.indexOf("/" + model + "/") > 0)
                    && (path.indexOf("/" + group + "/") > 0) &&
                    (path.indexOf("/" + language + "/") > 0) &&
                    (path.indexOf("/" + skill + ".txt") > 0)) {
                skillMetadata.put("descriptions", entry.getValue());
            }
        }
   JSONObject json = new JSONObject(true);
        json.put("skill_metadata", skillMetadata);
        json.put("accepted", true);
        json.put("message", "Success: Fetched Skill's Metadata");
        return new ServiceResponse(json);

We can get the required parameters through call.get() method where first parameter is the key for which we want to get the value and second parameter is the default value. If the path contains the desired language, group and model and skill, we return it as a response containing the respective field. The endpoint can be tested http://127.0.0.1:4000/cms/getSkillMetadata.json?group=general&model=entertainment&language=en&skill=cat and you will get the following response.

This is how getSKillMetadata service works. To add the metadata  to the skill visit susi_skill_data, the storage place for susi skills. For more information and complete code take a look at Susi server and join gitter chat channel for discussions.

Resources

  • Read more about Servets – https://www.tutorialspoint.com/servlets/
  • The source code –https://github.com/fossasia/susi_server/blob/development/src/ai/susi/server/api/cms/
  • Read about servlet interface – https://www.javatpoint.com/Servlet-interface
Continue ReadingAdding Skill Metadata in SUSI AI
  • FOSSASIA
  • Blog
  • GitHub
  • Projects
  • Code of Conduct
  • About
  • Contact
Copyright - OceanWP Theme by OceanWP
 

Loading Comments...
 

You must be logged in to post a comment.