Adding New Arrivals in the Metrics of SUSI.AI

The SUSI Skill CMS homepage contains a lot of metics on the homepage. For example, the highest rated skills, latest skills, most used skills etc. Another important metric is the newly arrived skills in a given period of time, say in last week. This keeps the users updated with the system and allows them to know what work is going on the assistant. This also inspires the skill creators to create more skills.

Update skill listing API

To get the list of recently added skill, first of all, we need a mechanism to sort them in descending order of their creation time. Update the skill listing API ie, ListSkillService.java to sort skills by their creation time. The creation time is stored in “YYYY-MM-DD T M S” format, for ex “2018-08-12T03:11:32Z”. So it can be sorted easily using string comparison function.

Collections.sort(jsonValues, new Comparator<JSONObject>() {
    private static final String KEY_NAME = "creationTime";
    @Override
    public int compare(JSONObject a, JSONObject b) {
        String valA = new String();
        String valB = new String();
        int result = 0;
         try {
            valA = a.get(KEY_NAME).toString();
            valB = b.get(KEY_NAME).toString();
            result = valB.compareToIgnoreCase(valA);
        } catch (JSONException e) {
            e.printStackTrace();
        }
        return result;
    }
});

After sorting the skills in descending order of their creation date. We need to filter them based on the time period. For example, if the skills created in last days are required then we need a generalized filter for that. This can be achieved by creating a variable for the starting date of the required time period. Say, if the skill created in last 7 days are required, then the number of milliseconds equivalent to 7 days is subtracted from the current timestamp. All the skills created after this timestamp are added to the result while others are skipped.

if (dateFilter) {
	long durationInMillisec = TimeUnit.DAYS.toMillis(duration);
	long timestamp = System.currentTimeMillis() - durationInMillisec;
	String startDate = new Timestamp(timestamp).toString().substring(0, 10); //substring is used for getting timestamp upto date only
	String skillCreationDate = jsonValues.get(i).get("creationTime").toString().substring(0,10);
	if (skillCreationDate.compareToIgnoreCase(startDate) < 0)
	{
	 continue;
	}
}

This filtering works in the API only when the filter type is set to date and duration in days is passed in the endpoint.

Implement new arrivals on CMS

Create the MenuItems in the sidebar that shows the filter name and add onClick handler on them. The skill listing API with the duration filter is passed to the handler. 3 MenuItems are added:

  • Last 7 Days
  • Last 30 Days
  • Last 90 Days

<MenuItem  value="&applyFilter=true&filter_name=descending&filter_type=date&duration=7"  key="Last 7 Days"
  primaryText="Last 7 Days"
  onClick={event =>
    this.handleArrivalTimeChange(
      event, '&applyFilter=true&filter_name=descending&filter_type=date&duration=7',
    )
  }
/>

Create a handler that listens to the onClick event of the above MenuItems. This handler accepts the API endpoint and calls the loadCards function with it.

handleArrivalTimeChange = (event, value) => {
 this.setState({ filter: value }, function() {
   // console.log(this.state);
   this.loadCards();
 });
};

Resources

Continue Reading

Implementing the List View of the Skill Cards

In this blog post, we are going to understand the implementation of the UI for the SUSI.AI skill card that is displayed on various routes of the SUSI Skill CMS Web-App. Now, there are two types of views of the views for the skill cards – List view and Grid view. We will learn to implement the List View in this blog.

Final UI of the Skill Card

Going through the implementation

The UI has multiple components –

  • The image thumbnail.
  • The title and author section,
  • Below that we have examples, ratings and the description section.

Fetching the data

  • The Skill Metadata for each skill is passed as props from the parent of the component, where this UI is implemented. This data object contains the various data points that are needed to display the UI. The key values used are –
    • skill_name – Used in the Title of the Skill Card
    • image – Used to display the thumbnail image of the skill
    • model – used to create the link to the Skill Details page
    • group – used to create the link to the Skill Details page
    • language – used to create the link to the Skill Details page
    • skill_tag – used to create the link to the Skill Details page
    • examples – used to display the examples card.
    • author – used to display the Author name
    • skill_rating – Used to display the stars and the total number of ratings of the skill
  • The following image shows the various areas, where the data is being used.

Parsing the data and creating JSX

  • Below is the code used to parse the data and achieving the UI, followed by the explanation.

…..
loadSkillCards = () => {
  let cards = [];
  Object.keys(this.state.skills).forEach(el => {
    let skill = this.state.skills[el];
    let skill_name = 'Name not available', examples = [], image = '', description =      
    'No description available', author_name = 'Author', average_rating = 0, 
    total_rating = 0;
    if (skill.skill_name) 
      skill_name = skill.skill_name.charAt(0).toUpperCase() + skill_name.slice(1);
    ….
    // Similarly parse, image, descriptions, author 
    ….
    if (skill.examples)
      examples = skill.examples.slice(0, 2); // Select max 2 examples
    if (skill.skill_rating) {
      average_rating = parseFloat(skill.skill_rating.stars.avg_star);
      total_rating = parseInt(skill.skill_rating.stars.total_star, 10);
    }
    cards.push(
      <div style={styles.skillCard} key={el}>
        <div style={styles.imageContainer}>
          // Display the image, else default avatar compoennt CircleImage
        </div>
        <div style={styles.content}>
          <div style={styles.header}>
            // Add Link to the skill title
              <div style={styles.title}><span>{skill_name}</span></div>
            <div style={styles.authorName}><span>{author_name}</span></div>
          </div>
          <div style={styles.details}>
            <div style={styles.exampleSection}>
              {examples.map((eg, index) => { return (
                <div key={index} style={styles.example}>&quot;{eg}&quot;</div>);
              })}
            </div>
            <div style={styles.textData}>
              <div style={styles.row}>
                <div style={styles.rating}>
                  // Show the 5-star rating section
                </div>
              </div>
              <div style={styles.row}>
                // Insert the skill description
              </div>
        //Close the div tags
    );
  });
  this.setState({cards});
};

render() {
.
.
  return (<div style={styles.gridList}>{skillDisplay}</div>);
}
.
.

 

  • An array of skills is passed as props and set in the state of the component in the constructor lifecycle method. The loadSkillCards() function is called in the didComponentMount lifecycle method, which is responsible for creating the JSX for all the Skill Cards.
  • In this function, the map property of array is used, to iterate over each skill and the corresponding Skill Card is pushed to the cards array, after successfully parsing the data.
  • At the end of the function definition, the state is updated, which in turn triggers the render function. In the render function, the cards are returned enclosed in a <div> tag. This helps us to create the above UI.

Styling the UI

Some important styling used is shown below. For the full styles object, please follow this link.

const styles = {
  skillCard: {
    width: '100%',
    overflow: 'hidden',
    display: 'flex',
    flexDirection: 'row',
    borderTop: '1px solid #eaeded',
    padding: 7,
  },
  imageContainer: {
    display: 'inline-block',
    alignItems: 'center',
    padding: '10px',
    background: '#fff',
    height: '218px',
    marginBottom: '6px',
  },
  image: {
    position: 'relative',
    height: '180px',
    width: '180px',
    verticalAlign: 'top',
    borderRadius: '50%',
  },
….
  gridlist: {
    marginTop: '20px',
    marginBottom: '40px',
    padding: '0px 10px',
    width: '100%',
….
  example: {
    fontStyle: 'italic',
    fontSize: '14px',
    padding: '14px 18px',
    borderRadius: '4px',
    border: '1px #ddd solid',
    float: 'left',
    display: 'flex row',
    justifyContent: 'center',
    alignItems: 'center',
    width: 192,
  },
….
};

export default styles;

 

I hope the implementation of the UI is clear and proved to be helpful for your understanding.

Resources

Showcase of the Material-UI icons (used for View type icons) – https://material.io/icons/

Continue Reading

Adding Functionality to Switch between List and Grid View of the Skill Cards on SUSI.AI CMS

In this blog post, we are going to understand the implementation of the functionality that enables the user to switch between the List View and the Grid View UI for the skill cards that is displayed on various routes of the SUSI Skill CMS Web-App. Let us go through the implementation in the blog –

Working of the feature

Going through the implementation

  • The UI for implementing the switching of views was achieved via the use of RadioButtonGroup component of the Material-UI library for React.
  • The type of view currently being shown was stored in the component state of the BrowseSkill component as viewType, whose default value is set to list, indicating that the skills are firstly shown in a List View.

.
.

export default class BrowseSkill extends React.Component {
  constructor(props) {
    super(props);
    this.state = {
        .
        …..
        viewType: 'list',
        ….
    };
  }
….
}

 

  • The RadioButtonGroup component has 2 child components, for each view. The child component that is to be used is RadioButton.
  • The props passed in the RadioButtonGroup are –
    • name : It is the name given to the component.
    • defaultSelected : It is the default view type.
    • style : It contains the style object of the UI.
    • valueSelected : It is set to the state variable assigned for storing view type.
    • onChange : It is the handler which executes, when the radio buttons are clicked.
  • The style for the desktop view and mobile view is different depending on the screen size and is follows –

//Mobile view
Style {
    right: 12,
    position: 'absolute',
    top: 216,
    display: 'flex',
}

//Desktop view
style={
    display: 'flex',
    marginTop: 34
}

 

  • The props passed in the RadioButton are –
    • value : The value stored in the state, that is responsible for the view type. The values for List and Grid view are list and grid respectively.
    • label : The label for the RadioButton.
    • labelStyle : The style object for the label.
    • checkedIcon : The icon used in the checked state.
    • uncheckedIcon : The icon used in the unchecked state.

UI of the Radio Buttons

  • The onClick handler of the radio buttons is –

handleViewChange = (event, value) => {
    this.setState({ viewType: value });
};

 

  • The code snippet for the UI implementation, written inside the render function is as follows :

….
<RadioButtonGroup
  name="view_type"
  defaultSelected="list"
  style={
    window.innerWidth < 430
      ? {
          right: 12,
          position: 'absolute',
          top: 216,
          display: 'flex',
        }
      : { display: 'flex', marginTop: 34 }
  }
  valueSelected={this.state.viewType}
  onChange={this.handleViewChange}
>
  <RadioButton
    value="list"
    label="List view"
    labelStyle={{ display: 'none' }}
    style={{ width: 'fit-content' }}
    checkedIcon={
      <ActionViewStream style={{ fill: '#4285f4' }} />
    }
    uncheckedIcon={<ActionViewStream />}
  />
  <RadioButton
    value="grid"
    label="Grid view"
    labelStyle={{ display: 'none' }}
    style={{ width: 'fit-content' }}
    checkedIcon={
      <ActionViewModule style={{ fill: '#4285f4' }} />
    }
    uncheckedIcon={<ActionViewModule />}
  />
</RadioButtonGroup>
….

 

I hope the implementation of the switching between Views would be clear after going through the blog and proved to be helpful for your understanding.

References

Continue Reading

Adding a Horizontally scrollable component to display Skills based on metrics

In this blog post, I will discuss about the implementation of a horizontally scrollable component to display skill based on metrics. The purpose of the implementation is to show top skills based on metrics related to usage, ratings, etc in SUSI.AI Skills CMS.

Implementational details

  • We call this component SkillCardScrollList which takes in a list of cards to be displayed along with some other properties and returns an UI, as shown in the above GIF.
  • The parameters that the component takes are:
    • scrollId: It is a required field of the type String. It is the id name of the horizontally scrollable div.
    • skills: It contains an array of cards that are to displayed inside the container.
    • languageValue: It represents the language of the skills that are shown.
    • skillUrl: It contains the URL that the app would be taken to, on clicking individual Skill Card.
    • modelValue: It contains the model that the skill belongs to.
  • Here is a sample of how it is used in the BrowseSkill component, for showing the Top Rated Skills in a SkillCardsScrollList
<SkillCardScrollList
    scrollId="topRated"
    skills={this.state.topRatedSkills}
    modalValue={this.state.modalValue}
    languageValue={this.state.languageValue}
    skillUrl={this.state.skillUrl}
/>

 

  • The reason behind passing an unique scrollId as a prop to the component is that, there was a need to trigger the scroll event of the scrollable div n the click of left and right Floating Action Buttons (FABs) as shown in the UI. And, on multiple imports of this component, there would have been inconsistent scroll behaviour seen, had it not been unique.
  • Following in the code block of the component, which will be explained in details, that deals with the main implementation –
.
.
.
.
  scrollLeft = () => {
    let parentEle = document.getElementById(this.props.scrollId);
    let scrollValue = $(parentEle).scrollLeft() - 200;
    $(parentEle)
      .stop()
      .animate({ scrollLeft: scrollValue }, 100);
  };

  scrollRight = () => {
    // Similar function of scrollLeft
  };

  loadSkillCards = () => {
    let cards = [];
    Object.keys(this.state.skills).forEach(el => {
      .
      /* Each skill object is passed and then pushed to the cards
        array*/
      .
      );
    });
    // Set the cards array in the state 
    this.setState({
      cards,
    });
  };

  render() {
    return (
      <div
        style={{
          marginTop: '20px',
          marginBottom: '40px',
          textAlign: 'justify',
          fontSize: '0.1px',
          width: '100%',
        }}
      >
        <div>
          <div
            id={this.props.scrollId}
            className="scrolling-wrapper"
            style={styles.gridList}
          >
            <FloatingActionButton
              mini={true}
              backgroundColor={'#4285f4'}
              style={styles.leftFab}
              onClick={this.scrollLeft}
            >
              <NavigationChevronLeft />
            </FloatingActionButton>
            {this.state.cards}
            <FloatingActionButton
              mini={true}
              backgroundColor={'#4285f4'}
              style={styles.rightFab}
              onClick={this.scrollRight}
            >
              <NavigationChevronRight />
            </FloatingActionButton>
          </div>
        </div>
      </div>
    );
  }
}

 

  • The div with class scrolling-wrapper is actually scrolled on the click of the left and right FAB. For choosing the correct div to be scrolled, there was a necessary condition of an unique id as explained earlier, which has been set to the div.
  • For making the component horizontally scrollable, specific CSS rules are added to the div. They are –
gridList: {
  margin: '10px',
  textAlign: 'center',
  overflowX: 'scroll',
  overflowY: 'hidden',
  whiteSpace: 'nowrap',
},
leftFab: {
  position: 'absolute',
  left: 260,
  marginTop: 75,
},
rightFab: {
  position: 'absolute',
  right: 0,
  marginTop: 75,
  marginRight: 10,
},

 

  • The CSS rules for the FABs make them fixed in a position and only lets the card list scroll.
  • Lastly, there was a issue regarding the presence of horizontal scroll-bar been shown, which makes the UI look a bit unpleasant.

  • It was hidden with a pseudo selector CSS rule.
div.scrolling-wrapper::-webkit-scrollbar {
    display: none;
}

 

This was the implementation for the horizontally scrollable component for displaying Skill List based on a standard metrics. I hope, you found the blog helpful in making the understanding of the implementation better.

Resources

Continue Reading

How search works on Susi Skill CMS

The SUSI Skill CMS is a central dashboard where the developers can add skills to make SUSI more intelligent. One of the major thing that was missing there was a search bar. So I recently added one that can search a skill based on:

  • Skill name
  • Skill description
  • Author name
  • Examples of the skill

How to add the search bar?

  1. Install the Material Search Bar component from the terminal.

npm i --save material-ui-search-bar

 

  1. Import this component in the BrowseSkill.js file.

import SearchBar from 'material-ui-search-bar'

 

  1. Create a state variables for the search query and initialize it to an empty string.

this.state = {
	...
	searchQuery:''
	...
};

 

  1. Add the search bar (UI Part) below the filters on CMS. Add a listener function to it that is called when the value of the search query changes.

<SearchBar
     onChange={this.handleSearch}
     value={this.state.searchQuery}
   />

 

  1. The search handler : Create a handler (handleSearch) to that listens to onChange events on the SearchBar. This function sets the value of searhQuery state variable and loads the card again based on the search filter.

handleSearch
  = (value) => {
    this.setState({searchQuery: value}, function () {
    this.loadCards();
    });
};

 

  1. The loadCards() function : The loadCards() function send a request to the Susi Server which in turn send a json response. Then this function makes cards for every skill and adds them to the CMS. Modify the loadCards() function to filter the cards array based on the search query. The javascript string function match() is used to check if the skill name, description, author’s name or examples match the search query. The filter() function adds the skill card to the filtered data if the match() function returns true (i.e., the skill is relevant to the search query).

How the filter works?

  1. First check if there’s something to search for. If the searchQuery.length is equal to zero then that means that there is nothing to search for.

self.state.searchQuery.length >0

 

  1. Then filter the related results based on
  • Skill name : The filter() function adds the skill card to the filtered data if the match function returns true (i.e., the skill is relevant to the search query). The match() function retuns true if the skill name matches the search query.

if (i.skill_name) {
    	result =  i.skill_name.toLowerCase()
        	.match( self.state.searchQuery.toLowerCase() );
    	if (result) {
        	return result;
    	}
}

 

Similarly, filter the cards on the basis of skill description and skill author’s name.

  • Skill examples: Loop over all the skill examples and check if any example matches the search query.

if (i.examples && i.examples.length>0) {
    	i.examples.map((el,j)=>{
      	result =  el.toLowerCase()
        	.match( self.state.searchQuery.toLowerCase() );
      	if (result) {
          	return result;
      	}
    	})
}

Example

Here the search query is “country”. The word “country” appears the skill description of the filtered cards.

Resources

Continue Reading

Link Preview Service from SUSI Server

 SUSI Webchat, SUSI Android app, SUSI iOS app are various SUSI clients which depend on response from SUSI Server. The most common response of SUSI Server is in form of links. Clients usually need to show the preview of the links to the user. This preview may include featured image, description and the title of the link.  Clients show this information by using various 3rd party APIs and libraries. We planned to create an API endpoint for this on SUSI Server to give the preview of the link. This service is called LinkPreviewService.
String url = post.get("url", "");
        if(url==null || url.isEmpty()){
            jsonObject.put("message","URL Not given");
            jsonObject.put("accepted",false);
            return new ServiceResponse(jsonObject);
        }

This API Endpoint accept only 1 get parameter which is the URL whose preview is to be shown.

Here we also check if no parameter or wrong URL parameter was sent. If that was the the case then we return an error message to the user.

 SourceContent sourceContent =     TextCrawler.scrape(url,3);
        if (sourceContent.getImages() != null) jsonObject.put("image", sourceContent.getImages().get(0));
        if (sourceContent.getDescription() != null) jsonObject.put("descriptionShort", sourceContent.getDescription());
        if(sourceContent.getTitle()!=null)jsonObject.put("title", sourceContent.getTitle());
        jsonObject.put("accepted",true);
        return new ServiceResponse(jsonObject);
    }

The TextCrawler function accept two parameters. One is the url of the website which is to be scraped for the preview data and the other is depth. To get the images, description and title there are methods built in. Here we just call those methods and set them in our JSON Object.

 private String htmlDecode(String content) {
        return Jsoup.parse(content).text();
    }

Text Crawler is based on Jsoup. Jsoup is a java library that is used to scrape HTML pages.

To get anything from Jsoup we need to decode the content of HTML to Text.

public List<String> getImages(Document document, int imageQuantity) {
        Elements media = document.select("[src]");
        while(var5.hasNext()) {
            Element srcElement = (Element)var5.next();
            if(srcElement.tagName().equals("img")) {
                ((List)matches).add(srcElement.attr("abs:src"));
            }
        }

 The getImages method takes the HTML document from the JSoup and find the image tags in that. We have given the imageQuantity parameter in the function, so accordingly it returns the src attribute of the first n images it find.

This API Endpoint can be seen working on

http://127.0.0.1:4000/susi/linkPreview.json?url=<ANY URL>

A real working example of this endpoint would be http://api.susi.ai/susi/linkPreview.json?url=https://techcrunch.com/2017/07/23/dear-tech-dudes-stop-being-such-idiots-about-women/

Resources:

Web Crawlers: https://www.promptcloud.com/data-scraping-vs-data-crawling/

JSoup: https://jsoup.org/

JSoup Api Docs: https://jsoup.org/apidocs/

Parsing HTML with JSoup: http://www.baeldung.com/java-with-jsoup

Continue Reading

Modifying SUSI Skills using SUSI Skill CMS

SUSI Skill CMS is a complete solution right from creating a skill to modifying the skill. The skills in SUSI are well synced with the remote repository and can be completely modified using the Edit Skill feature of SUSI Skill CMS. Here’s how to Modify a Skill.

  1. Sign Up/Login to the website using your credentials in skills.susi.ai
  2. Choose the SKill which you want to edit and click on the pencil icon.
  3. The following screen allows editing the skill. One can change the Group, Language, Skill Name, Image and the content as well.
  4. After making the changes the commit message can be added to Save the changes.

To achieve the above steps we require the following API Endpoints of the SUSI Server.

  1. http://api.susi.ai/cms/getSkillMetadata.json – This gives us the meta data which populates the various Skill Content, Image, Author etc.
  2. http://api.susi.ai/cms/getAllLanguages.json – This gives us all the languages of a Skill Group.
  3. http://api.susi.ai/cms/getGroups.json – This gives us all the list of Skill Groups whether Knowledge, Entertainment, Smalltalk etc.

Now since we have all the APIs in place we make the following AJAX calls to update the Skill Process.

  1. Since we are detecting changes in all the fields (Group Value, Skill Name, Language Value, Image Value, Commit Message, Content changes and the format of the content), the AJAX call can only be sent when there is a change in the PR and there is no null or undefined value in them. For that, we make various form validations. They are as follows.
    1. We first detect whether the User is in a logged in state.
if (!cookies.get('loggedIn')) {
            notification.open({
                message: 'Not logged In',
                description: 'Please login and then try to create/edit a skill',
                icon: <Icon type="close-circle" style={{ color: '#f44336' }} />,
            });
        }
  1. We check whether the image uploaded matches the format of the Skill image to be stored which is ::image images/imageName.png
if (!new RegExp(/images\/\w+\.\w+/g).test(this.state.imageUrl)) {
            notification.open({
                message: 'Error Processing your Request',
                description: 'image must be in format of images/imageName.jpg',
                icon: <Icon type="close-circle" style={{ color: '#f44336' }} />,
            });
        }
  1. We check if the commit message is not null and notify the user if he forgot to add a message.
if (this.state.commitMessage === null) {
            notification.open({
                message: 'Please make some changes to save the Skill',
                icon: <Icon type="close-circle" style={{ color: '#f44336' }} />,
            });
        }
  1. We also check whether the old values of the skill are completely similar to the new ones, in this case, we do not send the request.
if (toldValues===newValues {
            notification.open({
                message: 'Please make some changes to save the Skill',
                icon: <Icon type="close-circle" style={{ color: '#f44336' }} />,
            });
        }

To check out the complete code, go to this link.

  1. Next, if the above validations are successful, we send a POST request to the server and show the notification to the user accordingly, whether the changes to the Skill Data have been updated or not. Here’s the AJAX call snippet.
// create a form object
let form = new FormData();       
/* Append the following fields from the Skill Component:- OldModel, OldLanguage, OldSkill, NewModel, NewGroup, NewLanguage, NewSkill, changelog, content, imageChanged, old_image_name, new_image_name, image_name_changed, access_token */  
if (image_name_changed) {
            file = this.state.file;
            // append file to image
        }

        let settings = {
            "async": true,
            "crossDomain": true,
            "url": "http://api.susi.ai/cms/modifySkill.json",
            "method": "POST",
            "processData": false,
            "contentType": false,
            "mimeType": "multipart/form-data",
            "data": form
        };
        $.ajax(settings)..done(function (response) {
         //show success
        }.
        .fail(function(response){
         // show failure
        }
  1. To verify all this we head to the commits section of the SUSI Skill Data repo and see the changes we made. The changes can be seen here https://github.com/fossasia/susi_skill_data/commits/master 

Resources

  1. AJAX POST Request – https://api.jquery.com/jquery.post/ 
  2. Material UI – http://material-ui.com 
  3. Notifications – http://www.material-ui.com/#/components/notification 
Continue Reading

Deleting SUSI Skills from Server

SUSI Skill CMS is a web application to create and edit skills. In this blog post I will be covering how we made the skill deleting feature in Skill CMS from the SUSI Server.
The deletion of skill was to be made in such a way that user can click a button to delete the skill. As soon as they click the delete button the skill is deleted it is removed from the directory of SUSI Skills. But admins have an option to recover the deleted skill before completion of 30 days of deleting the skill.

First we will accept all the request parameters from the GET request.

        String model_name = call.get("model", "general");
        String group_name = call.get("group", "Knowledge");
        String language_name = call.get("language", "en");
        String skill_name = call.get("skill", "wikipedia");

In this we get the model name, category, language name, skill name and the commit ID. The above 4 parameters are used to make a file path that is used to find the location of the skill in the Susi Skill Data repository.

 if(!DAO.deleted_skill_dir.exists()){
            DAO.deleted_skill_dir.mkdirs();
   }

We need to move the skill to a directory called deleted_skills_dir. So we check if the directory exists or not. If it not exists then we create a directory for the deleted skills.

  if (skill.exists()) {
   File file = new File(DAO.deleted_skill_dir.getPath()+path);
   file.getParentFile().mkdirs();
   if(skill.renameTo(file)){
   Boolean changed =  new File(DAO.deleted_skill_dir.getPath()+path).setLastModified(System.currentTimeMillis());
     }

This is the part where the real deletion happens. We get the path of the skill and rename that to a new path which is in the directory of deleted skills.

Also here change the last modified time of the skill as the current time. This time is used to check if the skill deleted is older than 30 days or not.

    try (Git git = DAO.getGit()) {
                DAO.pushCommit(git, "Deleted " + skill_name, rights.getIdentity().isEmail() ? rights.getIdentity().getName() : "[email protected]");
                json.put("accepted", true);
                json.put("message", "Deleted " + skill_name);
            } catch (IOException | GitAPIException e) {

Finally we add the changes to Git. DAO.pushCommit pushes to commit to the Susi Skill Data repository. If the user is logged in we get the email of the user and set that email as the commit author. Else we set the username “[email protected]”.

Then in the caretaker class there is a method deleteOldFiles that checks for all the files whose last modified time was older than 30 days. If there is any file whose last modified time was older than 30 days then it quietly delete the files.

public void deleteOldFiles() {
     Collection<File> filesToDelete = FileUtils.listFiles(new         File(DAO.deleted_skill_dir.getPath()),
     new 
(DateTime.now().withTimeAtStartOfDay().minusDays(30).toDate()),
            TrueFileFilter.TRUE);    // include sub dirs
        for (File file : filesToDelete) {
               boolean success = FileUtils.deleteQuietly(file);
            if (!success) {
                System.out.print("Deleted skill older than 30 days.");
            }
      }
}

To test this API endpoint, we need to call http://localhost:4000/cms/deleteSkill.txt?model=general&group=Knowledge&language=en&skill=<skill_name>

Resources

JGit Documentation: https://eclipse.org/jgit/documentation/

Commons IO: https://commons.apache.org/proper/commons-io/

Age Filter: https://commons.apache.org/proper/commons-io/javadocs/api-1.4/org/apache/commons/io/filefilter/AgeFileFilter.html

JGit User Guide: http://wiki.eclipse.org/JGit/User_Guide

JGit Repository access: http://www.codeaffine.com/2014/09/22/access-git-repository-with-jgit/

Continue Reading

Getting SUSI Skill at a Commit ID

Susi Skill CMS is a web app to edit and create new skills. We use Git for storing different versions of Susi Skills. So what if we want to roll back to a previous version of the skill? To implement this feature in Susi Skill CMS, we needed an API endpoint which accepts the name of the skill and the commit ID and returns the file at that commit ID.

In this blog post I will tell about making an API endpoint which works similar to git show.

First we will accept all the request parameters from the GET request.

        String model_name = call.get("model", "general");
        String group_name = call.get("group", "Knowledge");
        String language_name = call.get("language", "en");
        String skill_name = call.get("skill", "wikipedia");
        String commitID  = call.get("commitID", null);

In this we get the model name, category, language name, skill name and the commit ID. The above 4 parameters are used to make a file path that is used to find the location of the skill in the Susi Skill Data repository.

This servlet need CommitID to work and if commit ID is not given in the request parameters then we send an error message saying that the commit id is null and stop the servlet execution.

    Repository repository = DAO.getRepository();
    ObjectId CommitIdObject = repository.resolve(commitID);

Then we get the git repository of the skill from the DAO and initialize the repository object.

From the commitID that we got in the request parameters we create a CommitIdObject.

   (RevWalk revWalk = new RevWalk(repository)) {
   RevCommit commit = revWalk.parseCommit(CommitIdObject);
   RevTree tree = commit.getTree();


Now using commit’s tree, we will find the find the path and get the tree of the commit.

From the TreeWalk in the repository we will set a filter to find a file. This searches recursively for the files inside all the folders.

                revWalk = new RevWalk(repository)) {
                try (TreeWalk treeWalk = new TreeWalk(repository)) {
                    treeWalk.addTree(tree);
                    treeWalk.setRecursive(true);
                    treeWalk.setFilter(PathFilter.create(path));
                    if (!treeWalk.next()) {
                        throw new IllegalStateException("Did not find expected file");
                    }

If the TreeWalk reaches to an end and does not find the specified skill path then it returns anIllegal State Exception with an message saying did not found the file on that commit ID.

       ObjectId objectId = treeWalk.getObjectId(0);
       ObjectLoader loader = repository.open(objectId);
       OutputStream output = new OutputStream();
       loader.copyTo(output);

And then one can the loader to read the file. From the treeWalk we get the object and create an output stream to copy the file content in it. After that we create the JSON and put the OutputStream object as as String in it.

       json.put("file",output);

This Servlet can be seen working api.susi.ai: http://api.susi.ai/cms/getFileAtCommitID.json?model=general&group=knowledge&language=en&skill=bitcoin&commitID=214791f55c19f24d7744364495541b685539a4ee

Resources

JGit Documentation: https://eclipse.org/jgit/documentation/

JGit User Guide: http://wiki.eclipse.org/JGit/User_Guide

JGit Repository access: http://www.codeaffine.com/2014/09/22/access-git-repository-with-jgit/

JGit Github: https://github.com/eclipse/jgit

Continue Reading
Close Menu