How search works on Susi Skill CMS

The SUSI Skill CMS is a central dashboard where the developers can add skills to make SUSI more intelligent. One of the major thing that was missing there was a search bar. So I recently added one that can search a skill based on:

  • Skill name
  • Skill description
  • Author name
  • Examples of the skill

How to add the search bar?

  1. Install the Material Search Bar component from the terminal.

npm i --save material-ui-search-bar

 

  1. Import this component in the BrowseSkill.js file.

import SearchBar from 'material-ui-search-bar'

 

  1. Create a state variables for the search query and initialize it to an empty string.

this.state = {
	...
	searchQuery:''
	...
};

 

  1. Add the search bar (UI Part) below the filters on CMS. Add a listener function to it that is called when the value of the search query changes.

<SearchBar
     onChange={this.handleSearch}
     value={this.state.searchQuery}
   />

 

  1. The search handler : Create a handler (handleSearch) to that listens to onChange events on the SearchBar. This function sets the value of searhQuery state variable and loads the card again based on the search filter.

handleSearch
  = (value) => {
    this.setState({searchQuery: value}, function () {
    this.loadCards();
    });
};

 

  1. The loadCards() function : The loadCards() function send a request to the Susi Server which in turn send a json response. Then this function makes cards for every skill and adds them to the CMS. Modify the loadCards() function to filter the cards array based on the search query. The javascript string function match() is used to check if the skill name, description, author’s name or examples match the search query. The filter() function adds the skill card to the filtered data if the match() function returns true (i.e., the skill is relevant to the search query).

How the filter works?

  1. First check if there’s something to search for. If the searchQuery.length is equal to zero then that means that there is nothing to search for.

self.state.searchQuery.length >0

 

  1. Then filter the related results based on
  • Skill name : The filter() function adds the skill card to the filtered data if the match function returns true (i.e., the skill is relevant to the search query). The match() function retuns true if the skill name matches the search query.

if (i.skill_name) {
    	result =  i.skill_name.toLowerCase()
        	.match( self.state.searchQuery.toLowerCase() );
    	if (result) {
        	return result;
    	}
}

 

Similarly, filter the cards on the basis of skill description and skill author’s name.

  • Skill examples: Loop over all the skill examples and check if any example matches the search query.

if (i.examples && i.examples.length>0) {
    	i.examples.map((el,j)=>{
      	result =  el.toLowerCase()
        	.match( self.state.searchQuery.toLowerCase() );
      	if (result) {
          	return result;
      	}
    	})
}

Example

Here the search query is “country”. The word “country” appears the skill description of the filtered cards.

Resources

Link Preview Service from SUSI Server

 SUSI Webchat, SUSI Android app, SUSI iOS app are various SUSI clients which depend on response from SUSI Server. The most common response of SUSI Server is in form of links. Clients usually need to show the preview of the links to the user. This preview may include featured image, description and the title of the link.  Clients show this information by using various 3rd party APIs and libraries. We planned to create an API endpoint for this on SUSI Server to give the preview of the link. This service is called LinkPreviewService.
String url = post.get("url", "");
        if(url==null || url.isEmpty()){
            jsonObject.put("message","URL Not given");
            jsonObject.put("accepted",false);
            return new ServiceResponse(jsonObject);
        }

This API Endpoint accept only 1 get parameter which is the URL whose preview is to be shown.

Here we also check if no parameter or wrong URL parameter was sent. If that was the the case then we return an error message to the user.

 SourceContent sourceContent =     TextCrawler.scrape(url,3);
        if (sourceContent.getImages() != null) jsonObject.put("image", sourceContent.getImages().get(0));
        if (sourceContent.getDescription() != null) jsonObject.put("descriptionShort", sourceContent.getDescription());
        if(sourceContent.getTitle()!=null)jsonObject.put("title", sourceContent.getTitle());
        jsonObject.put("accepted",true);
        return new ServiceResponse(jsonObject);
    }

The TextCrawler function accept two parameters. One is the url of the website which is to be scraped for the preview data and the other is depth. To get the images, description and title there are methods built in. Here we just call those methods and set them in our JSON Object.

 private String htmlDecode(String content) {
        return Jsoup.parse(content).text();
    }

Text Crawler is based on Jsoup. Jsoup is a java library that is used to scrape HTML pages.

To get anything from Jsoup we need to decode the content of HTML to Text.

public List<String> getImages(Document document, int imageQuantity) {
        Elements media = document.select("[src]");
        while(var5.hasNext()) {
            Element srcElement = (Element)var5.next();
            if(srcElement.tagName().equals("img")) {
                ((List)matches).add(srcElement.attr("abs:src"));
            }
        }

 The getImages method takes the HTML document from the JSoup and find the image tags in that. We have given the imageQuantity parameter in the function, so accordingly it returns the src attribute of the first n images it find.

This API Endpoint can be seen working on

http://127.0.0.1:4000/susi/linkPreview.json?url=<ANY URL>

A real working example of this endpoint would be http://api.susi.ai/susi/linkPreview.json?url=https://techcrunch.com/2017/07/23/dear-tech-dudes-stop-being-such-idiots-about-women/

Resources:

Web Crawlers: https://www.promptcloud.com/data-scraping-vs-data-crawling/

JSoup: https://jsoup.org/

JSoup Api Docs: https://jsoup.org/apidocs/

Parsing HTML with JSoup: http://www.baeldung.com/java-with-jsoup

Modifying SUSI Skills using SUSI Skill CMS

SUSI Skill CMS is a complete solution right from creating a skill to modifying the skill. The skills in SUSI are well synced with the remote repository and can be completely modified using the Edit Skill feature of SUSI Skill CMS. Here’s how to Modify a Skill.

  1. Sign Up/Login to the website using your credentials in skills.susi.ai
  2. Choose the SKill which you want to edit and click on the pencil icon.
  3. The following screen allows editing the skill. One can change the Group, Language, Skill Name, Image and the content as well.
  4. After making the changes the commit message can be added to Save the changes.

To achieve the above steps we require the following API Endpoints of the SUSI Server.

  1. http://api.susi.ai/cms/getSkillMetadata.json – This gives us the meta data which populates the various Skill Content, Image, Author etc.
  2. http://api.susi.ai/cms/getAllLanguages.json – This gives us all the languages of a Skill Group.
  3. http://api.susi.ai/cms/getGroups.json – This gives us all the list of Skill Groups whether Knowledge, Entertainment, Smalltalk etc.

Now since we have all the APIs in place we make the following AJAX calls to update the Skill Process.

  1. Since we are detecting changes in all the fields (Group Value, Skill Name, Language Value, Image Value, Commit Message, Content changes and the format of the content), the AJAX call can only be sent when there is a change in the PR and there is no null or undefined value in them. For that, we make various form validations. They are as follows.
    1. We first detect whether the User is in a logged in state.
if (!cookies.get('loggedIn')) {
            notification.open({
                message: 'Not logged In',
                description: 'Please login and then try to create/edit a skill',
                icon: <Icon type="close-circle" style={{ color: '#f44336' }} />,
            });
        }
  1. We check whether the image uploaded matches the format of the Skill image to be stored which is ::image images/imageName.png
if (!new RegExp(/images\/\w+\.\w+/g).test(this.state.imageUrl)) {
            notification.open({
                message: 'Error Processing your Request',
                description: 'image must be in format of images/imageName.jpg',
                icon: <Icon type="close-circle" style={{ color: '#f44336' }} />,
            });
        }
  1. We check if the commit message is not null and notify the user if he forgot to add a message.
if (this.state.commitMessage === null) {
            notification.open({
                message: 'Please make some changes to save the Skill',
                icon: <Icon type="close-circle" style={{ color: '#f44336' }} />,
            });
        }
  1. We also check whether the old values of the skill are completely similar to the new ones, in this case, we do not send the request.
if (toldValues===newValues {
            notification.open({
                message: 'Please make some changes to save the Skill',
                icon: <Icon type="close-circle" style={{ color: '#f44336' }} />,
            });
        }

To check out the complete code, go to this link.

  1. Next, if the above validations are successful, we send a POST request to the server and show the notification to the user accordingly, whether the changes to the Skill Data have been updated or not. Here’s the AJAX call snippet.
// create a form object
let form = new FormData();       
/* Append the following fields from the Skill Component:- OldModel, OldLanguage, OldSkill, NewModel, NewGroup, NewLanguage, NewSkill, changelog, content, imageChanged, old_image_name, new_image_name, image_name_changed, access_token */  
if (image_name_changed) {
            file = this.state.file;
            // append file to image
        }

        let settings = {
            "async": true,
            "crossDomain": true,
            "url": "http://api.susi.ai/cms/modifySkill.json",
            "method": "POST",
            "processData": false,
            "contentType": false,
            "mimeType": "multipart/form-data",
            "data": form
        };
        $.ajax(settings)..done(function (response) {
         //show success
        }.
        .fail(function(response){
         // show failure
        }
  1. To verify all this we head to the commits section of the SUSI Skill Data repo and see the changes we made. The changes can be seen here https://github.com/fossasia/susi_skill_data/commits/master 

Resources

  1. AJAX POST Request – https://api.jquery.com/jquery.post/ 
  2. Material UI – http://material-ui.com 
  3. Notifications – http://www.material-ui.com/#/components/notification 

Deleting SUSI Skills from Server

SUSI Skill CMS is a web application to create and edit skills. In this blog post I will be covering how we made the skill deleting feature in Skill CMS from the SUSI Server.
The deletion of skill was to be made in such a way that user can click a button to delete the skill. As soon as they click the delete button the skill is deleted it is removed from the directory of SUSI Skills. But admins have an option to recover the deleted skill before completion of 30 days of deleting the skill.

First we will accept all the request parameters from the GET request.

        String model_name = call.get("model", "general");
        String group_name = call.get("group", "Knowledge");
        String language_name = call.get("language", "en");
        String skill_name = call.get("skill", "wikipedia");

In this we get the model name, category, language name, skill name and the commit ID. The above 4 parameters are used to make a file path that is used to find the location of the skill in the Susi Skill Data repository.

 if(!DAO.deleted_skill_dir.exists()){
            DAO.deleted_skill_dir.mkdirs();
   }

We need to move the skill to a directory called deleted_skills_dir. So we check if the directory exists or not. If it not exists then we create a directory for the deleted skills.

  if (skill.exists()) {
   File file = new File(DAO.deleted_skill_dir.getPath()+path);
   file.getParentFile().mkdirs();
   if(skill.renameTo(file)){
   Boolean changed =  new File(DAO.deleted_skill_dir.getPath()+path).setLastModified(System.currentTimeMillis());
     }

This is the part where the real deletion happens. We get the path of the skill and rename that to a new path which is in the directory of deleted skills.

Also here change the last modified time of the skill as the current time. This time is used to check if the skill deleted is older than 30 days or not.

    try (Git git = DAO.getGit()) {
                DAO.pushCommit(git, "Deleted " + skill_name, rights.getIdentity().isEmail() ? rights.getIdentity().getName() : "[email protected]");
                json.put("accepted", true);
                json.put("message", "Deleted " + skill_name);
            } catch (IOException | GitAPIException e) {

Finally we add the changes to Git. DAO.pushCommit pushes to commit to the Susi Skill Data repository. If the user is logged in we get the email of the user and set that email as the commit author. Else we set the username “[email protected]”.

Then in the caretaker class there is a method deleteOldFiles that checks for all the files whose last modified time was older than 30 days. If there is any file whose last modified time was older than 30 days then it quietly delete the files.

public void deleteOldFiles() {
     Collection<File> filesToDelete = FileUtils.listFiles(new         File(DAO.deleted_skill_dir.getPath()),
     new 
(DateTime.now().withTimeAtStartOfDay().minusDays(30).toDate()),
            TrueFileFilter.TRUE);    // include sub dirs
        for (File file : filesToDelete) {
               boolean success = FileUtils.deleteQuietly(file);
            if (!success) {
                System.out.print("Deleted skill older than 30 days.");
            }
      }
}

To test this API endpoint, we need to call http://localhost:4000/cms/deleteSkill.txt?model=general&group=Knowledge&language=en&skill=<skill_name>

Resources

JGit Documentation: https://eclipse.org/jgit/documentation/

Commons IO: https://commons.apache.org/proper/commons-io/

Age Filter: https://commons.apache.org/proper/commons-io/javadocs/api-1.4/org/apache/commons/io/filefilter/AgeFileFilter.html

JGit User Guide: http://wiki.eclipse.org/JGit/User_Guide

JGit Repository access: http://www.codeaffine.com/2014/09/22/access-git-repository-with-jgit/

Getting SUSI Skill at a Commit ID

Susi Skill CMS is a web app to edit and create new skills. We use Git for storing different versions of Susi Skills. So what if we want to roll back to a previous version of the skill? To implement this feature in Susi Skill CMS, we needed an API endpoint which accepts the name of the skill and the commit ID and returns the file at that commit ID.

In this blog post I will tell about making an API endpoint which works similar to git show.

First we will accept all the request parameters from the GET request.

        String model_name = call.get("model", "general");
        String group_name = call.get("group", "Knowledge");
        String language_name = call.get("language", "en");
        String skill_name = call.get("skill", "wikipedia");
        String commitID  = call.get("commitID", null);

In this we get the model name, category, language name, skill name and the commit ID. The above 4 parameters are used to make a file path that is used to find the location of the skill in the Susi Skill Data repository.

This servlet need CommitID to work and if commit ID is not given in the request parameters then we send an error message saying that the commit id is null and stop the servlet execution.

    Repository repository = DAO.getRepository();
    ObjectId CommitIdObject = repository.resolve(commitID);

Then we get the git repository of the skill from the DAO and initialize the repository object.

From the commitID that we got in the request parameters we create a CommitIdObject.

   (RevWalk revWalk = new RevWalk(repository)) {
   RevCommit commit = revWalk.parseCommit(CommitIdObject);
   RevTree tree = commit.getTree();


Now using commit’s tree, we will find the find the path and get the tree of the commit.

From the TreeWalk in the repository we will set a filter to find a file. This searches recursively for the files inside all the folders.

                revWalk = new RevWalk(repository)) {
                try (TreeWalk treeWalk = new TreeWalk(repository)) {
                    treeWalk.addTree(tree);
                    treeWalk.setRecursive(true);
                    treeWalk.setFilter(PathFilter.create(path));
                    if (!treeWalk.next()) {
                        throw new IllegalStateException("Did not find expected file");
                    }

If the TreeWalk reaches to an end and does not find the specified skill path then it returns anIllegal State Exception with an message saying did not found the file on that commit ID.

       ObjectId objectId = treeWalk.getObjectId(0);
       ObjectLoader loader = repository.open(objectId);
       OutputStream output = new OutputStream();
       loader.copyTo(output);

And then one can the loader to read the file. From the treeWalk we get the object and create an output stream to copy the file content in it. After that we create the JSON and put the OutputStream object as as String in it.

       json.put("file",output);

This Servlet can be seen working api.susi.ai: http://api.susi.ai/cms/getFileAtCommitID.json?model=general&group=knowledge&language=en&skill=bitcoin&commitID=214791f55c19f24d7744364495541b685539a4ee

Resources

JGit Documentation: https://eclipse.org/jgit/documentation/

JGit User Guide: http://wiki.eclipse.org/JGit/User_Guide

JGit Repository access: http://www.codeaffine.com/2014/09/22/access-git-repository-with-jgit/

JGit Github: https://github.com/eclipse/jgit

Implementation of Child Routes in SUSI Skill CMS

In a previous blog post I discussed about how we implemented routing in SUSI Web Chat Application. In this post I’m planning to discuss about how we developed child routes in SUSI Skill CMS .

When we start developing our application, it was working correctly but  all skills loaded in the same URL. ( skill.susi.ai/SkillPage ). When user clicks the edit button every skill loaded in the same URL ( skill.susi.ai/EditSkill ). We got a requirement to load each of our skills in separate routes. This is how we implemented the child routes of the application.

We wanted to show each individual skill under this type of URL,

skill.susi.ai/ [SKILL GROUP] / [SKILL NAME] / [LANGUAGE]

When user clicks on the edit button, we needed to show that particular skill under this URL.

skill.susi.ai/ [SKILL GROUP] / [SKILL NAME] / edit / [LANGUAGE]

First we set our routings in index.js file.

<Switch>
    <Route exact path="/:category/:skill/edit/:lang" component={Home} />
    <Route exact path="/:category/:skill/:lang" component={SkillListing}/>
    <Route exact path="/" component={BrowseSkill} />
    <Route exact path="*" component={NotFound} />
</Switch>

We have to add the “exact” attribute, if we don’t add that it will not redirect users to “404” page when user trying to access wrong routes.

Next step is sending data from one component to another component.
In SUSI Skill CMS, user can choose any skill from the home page. Then after it goes to the skill page and shows details about the selected skill. We have to modify the button as,

<Link to={{ pathname: '/'+self.state.groupValue+'/'+el+'/'+self.state.languageValue }} >
<Card>
</Card>
</Link>

Now the user clicks on the card. It changes the URL and loads the corresponding component according to the routes that we defined in “index.js” file previously.
Second thing that we need to do is to catch URL routs and render relevant data according to the URL routes.
Let’s say I clicked on “distance” skill. Then user will go to this URL “http://skills.susi.ai/Knowledge/distance/en ”
Now It loads the “SkillListing” component according to the route we defined in “ index.js ” here ””.
To derive data from URL we simply used these codes in “SkillListing.js”.

let baseUrl = 'http://api.susi.ai/cms/getSkillMetadata.json';           
let modelValue = "general";
           this.name = this.props.location.pathname.split('/')[2];
           this.groupValue = this.props.location.pathname.split('/')[1];
           this.languageValue = this.props.location.pathname.split('/')[3];
           url = baseUrl + '?model=' + modelValue + '&group=' + this.groupValue + '&language=' + this.languageValue + '&skill=' + this.name;

We collected data from the URL and made another URL, we used this URL to get details of the skill from the server. We used this urls as below.

           $.ajax({
               url: url,
               jsonpCallback: 'pc',
               dataType: 'jsonp',
               jsonp: 'callback',
               crossDomain: true,
               success: function (data) {
                   self.updateData(data.skill_metadata)
               }
           });

If the Ajax request is success, those data are passed to “updateData()” and it updates the component and shows to users like this.

We applied same mechanism to the edit button and edit page. This is how we modified skill.susi.ai ‘s Routings. If you like to contribute SUSI Skill CMS please fork our repository on github. here

Resources:

  • Previous Blogpost about routing: https://blog.fossasia.org/implementation-of-react-routers-in-susi-web-chat/
  • React Router v4 tutorial https://medium.com/@pshrmn/a-simple-react-router-v4-tutorial-7f23ff27adf

Enhanced Skill Tiles in SUSI Skill CMS

The SUSI Skill Wiki is a management system for all the SUSI Skills and the Skill display screen ought to look attractive. The earlier version of the Skill Display was just a display with the skill name populated as cards as shown in the image.

image

So as we progressed over to add more metadata to the SUSI Skills, we had the challenge to show all details which were as follows –

An example of a skill metadata format-

"cricketTest": {
      "image": "images/images.jpg",
      "author_url": "skill.susi.ai",
      "examples": ["Testing Works"],
      "developer_privacy_policy": "na",
      "author": "cms",
      "skill_name": "cricket",
      "dynamic_content": true,
      "terms_of_use": "na",
      "descriptions": "testing",
      "skill_rating": null
    }

To embed the Skill metadata in the Tiles the following steps are to be followed –

  1. We first use the end point at the SUSI Server, http://api.susi.ai/cms/getSkillList.json with the following attributes –
    1. model – The skill follows a general model or maybe a tutorial model
    2. group -The category or group of the skill.
    3. language – The language of the skill output.

2.  An AJAX request to this end point gives us the following response.

{
accepted: true,
model: "general",
group: "Knowledge",
language: "en",
skills: 
{
capital: 
{
image: "images/capital.png",
author_url: "https://github.com/chashmeetsingh",
examples: 
[
"capital of Bangladesh"
],
developer_privacy_policy: null,
author: "chashmeet singh",
skill_name: "capital",
dynamic_content: true,
terms_of_use: null,
descriptions: "a skill to tell user about capital of any country.",
skill_rating: 
{
negative: "0",
positive: "1"
}
}

We use the descriptions, skill_name, examples, image from the skill metadata to create our Skill Tile.

  1. The styles of the cards follow a CSS flexbox structure. A sample mock up of the Skill Card looks as follows.

We first handle all the base cases and show “No name available”, “No description available” for data which does not exist or is found to be “null”. We then create the card mock-up in ReactJS which looks somewhat like this code snippet in the file BrowseSkill.js

                            <Card style={styles.row} key={el}>
                                <div style={styles.right} key={el}>
                                    {image ? <div style={styles.imageContainer}>
                                        <img alt={skill_name} src={image} style={styles.image}/>
                                    </div>:
                                    <CircleImage name={el} size="48"/>}
                                    <div style={styles.titleStyle}>"{examples}"</div>
                                </div>
                                <div style={styles.details}>
                                    <h3 style={styles.name}>{skill_name}</h3>
                                    <p style={styles.description}>{description}</p>
                                </div>
                            </Card>

  1. We then add the following styles to the Card and its contents which complete the look of the View.
imageContainer:{   
        position: 'relative',
        height: '80px',
        width: '80px',
        verticalAlign: 'top'
    },
    name:{
        textAlign: 'left',
        fontSize: '15px',
        color: '#4285f4',
        margin: '4px 0'
    },
    details:{
        paddingLeft:'10px'
    },
    image:{
        maxWidth: '100%',
        border: 0
    },
description:{
        textAlign:'left',
        fontSize: '14px'   
    },
row: {
        width: 280,
        minHeight:'200px',
        margin:"10px",
        overflow:'hidden',
        justifyContent: "center",
        fontSize: '10px',
        textAlign: 'center',
        display: 'inline-block',
        background: '#f2f2f2',
        borderRadius: '5px',
        backgroundColor: '#f4f6f6',
        border: '1px solid #eaeded',
        padding: '4px',
        alignSelf:'flex-start'
    },
titleStyle:{
    textAlign: 'left',
    fontStyle: 'italic',
    fontSize: '16px',
    textOverflow: 'ellipsis',
    overflow: 'hidden',
    width: '138px',
    marginLeft: '15px',
    verticalAlign: 'middle',
    display: 'block'
}

To see the SUSI Skills or to contribute to it, please visit http://skills.susi.ai

Resources