Link Preview Service from SUSI Server

 SUSI Webchat, SUSI Android app, SUSI iOS app are various SUSI clients which depend on response from SUSI Server. The most common response of SUSI Server is in form of links. Clients usually need to show the preview of the links to the user. This preview may include featured image, description and the title of the link.  Clients show this information by using various 3rd party APIs and libraries. We planned to create an API endpoint for this on SUSI Server to give the preview of the link. This service is called LinkPreviewService.
String url = post.get("url", "");
        if(url==null || url.isEmpty()){
            jsonObject.put("message","URL Not given");
            jsonObject.put("accepted",false);
            return new ServiceResponse(jsonObject);
        }

This API Endpoint accept only 1 get parameter which is the URL whose preview is to be shown.

Here we also check if no parameter or wrong URL parameter was sent. If that was the the case then we return an error message to the user.

 SourceContent sourceContent =     TextCrawler.scrape(url,3);
        if (sourceContent.getImages() != null) jsonObject.put("image", sourceContent.getImages().get(0));
        if (sourceContent.getDescription() != null) jsonObject.put("descriptionShort", sourceContent.getDescription());
        if(sourceContent.getTitle()!=null)jsonObject.put("title", sourceContent.getTitle());
        jsonObject.put("accepted",true);
        return new ServiceResponse(jsonObject);
    }

The TextCrawler function accept two parameters. One is the url of the website which is to be scraped for the preview data and the other is depth. To get the images, description and title there are methods built in. Here we just call those methods and set them in our JSON Object.

 private String htmlDecode(String content) {
        return Jsoup.parse(content).text();
    }

Text Crawler is based on Jsoup. Jsoup is a java library that is used to scrape HTML pages.

To get anything from Jsoup we need to decode the content of HTML to Text.

public List<String> getImages(Document document, int imageQuantity) {
        Elements media = document.select("[src]");
        while(var5.hasNext()) {
            Element srcElement = (Element)var5.next();
            if(srcElement.tagName().equals("img")) {
                ((List)matches).add(srcElement.attr("abs:src"));
            }
        }

 The getImages method takes the HTML document from the JSoup and find the image tags in that. We have given the imageQuantity parameter in the function, so accordingly it returns the src attribute of the first n images it find.

This API Endpoint can be seen working on

http://127.0.0.1:4000/susi/linkPreview.json?url=<ANY URL>

A real working example of this endpoint would be http://api.susi.ai/susi/linkPreview.json?url=https://techcrunch.com/2017/07/23/dear-tech-dudes-stop-being-such-idiots-about-women/

Resources:

Web Crawlers: https://www.promptcloud.com/data-scraping-vs-data-crawling/

JSoup: https://jsoup.org/

JSoup Api Docs: https://jsoup.org/apidocs/

Parsing HTML with JSoup: http://www.baeldung.com/java-with-jsoup

Continue ReadingLink Preview Service from SUSI Server

Burst Camera Mode in Phimpme Android

Camera is an integral part of core feature in Phimpme Android. Various features were added in the camera part such as resolution, timer, shutter sound, white balance etc. Click burst shot from camera is also an important feature to be added. Burst shot is clicking multiple pictures in one go.

Adding a Burst mode in Phimpme Camera

  • Adding burst mode enable entry in options

The popup view in Camera is added programmatically in app. Setting up the values from sharedpreferences. It takes the value and set burst mode off, 1x, 2x etc. according to value.

final String[] burst_mode_values = getResources().getStringArray(R.array.preference_burst_mode_values);
  String[] burst_mode_entries = getResources().getStringArray(R.array.preference_burst_mode_entries);
String burst_mode_value = sharedPreferences.getString(PreferenceKeys.getBurstModePreferenceKey(), "1");

Two methods created for setting up the previous and next values. To set up the previous value we need to check the current value to be not equal to -1 and greater that zero. Upgrade or downgrade the value of burst mode, according to the click.

public int onClickPrev() {
         if( burst_mode_index != -1 && burst_mode_index > 0 ) {
            burst_mode_index--;
            update(); ...
}

public int onClickNext() {
            if( burst_mode_index != -1 && burst_mode_index < burst_mode_values.length-1 ) {
              burst_mode_index++;
            update();...
}
  • Saving the value in sharedpreferences

So on clicking the previous and next, the value of burst mode value will be updated. As shown in the above code snippet, after every increment and decrement the values set on view and called update method to update the value in the sharedpreference as shown below.

private void update() {
        String new_burst_mode_value = burst_mode_values[burst_mode_index];
        SharedPreferences sharedPreferences = PreferenceManager.getDefaultSharedPreferences(main_activity);
SharedPreferences.Editor editor = sharedPreferences.edit();
editor.putString(PreferenceKeys.getBurstModePreferenceKey(), new_burst_mode_value);
editor.apply();}

  • Taking multiple Images

Now in the implementation part, we need to continuously click the image according to the burst value set by the user. So to enable this, first check the value not to be negative and should be greater than zero. Whole iteration work on separate variable named remaining burst photos. The value of the variable decrease after every image click i.e. takePhoto method calls.

if( remaining_burst_photos == -1 || remaining_burst_photos > 0 ) {
  if( remaining_burst_photos > 0 )
     remaining_burst_photos--;
  long timer_delay = applicationInterface.getRepeatIntervalPref();
  if( timer_delay == 0 ) {
     phase = PHASE_TAKING_PHOTO;
     takePhoto(true);
  }
  else {
     takePictureOnTimer(timer_delay, true);
  }
}

Resources:

 

Continue ReadingBurst Camera Mode in Phimpme Android

Implementing Change Password Feature in SUSI Android App using Custom Dialogs

Recently a new servlet was implemented on the SUSI Server about changing the password of the logged in user. This feature comes in handy to avoid unauthorized usage of the SUSI Account. Almost all the online platforms have this feature to change the password to avoid notorious user to unethical use someone else’s account. In SUSI Android app this new API was used with a nice UI to change the password of the user. The process is very simple and easy to grasp. This blog will try to cover the API information and implementation of the Change Password feature in the android client.

API Information

For changing the password of SUSI Account of the user, we have to call on  /aaa/changepassword.json

We have to provide three parameters along with this api call:

  1. changepassword:  Email of user (type string) using which user is logged in.
  2. password:  Old password (type string with min length of 6) of the user.
  3. newpassword: New password (type string with min length of 6) of the user.
  4. access_token: An encrypted access_token indicating user is logged in.

Sample Response (Success)

{
  "session": {"identity": {
    "type": "email",
    "name": "YOUR_EMAIL_ADDRESS",
    "anonymous": false
  }},
  "accepted": true,
  "message": "Your password has been changed!"
}

Error Response (Failure). This happens when user is not logged in:

HTTP ERROR 401
Problem accessing /aaa/changepassword.json. Reason:
   Base user role not sufficient. Your base user role is 'ANONYMOUS', your user role is 'anonymous'

Implementation in SUSI Android App

The change password option is located in Settings Activity and displayed only when user is logged in. So, if a logged in user wants to change the password of his/her SUSI AI account, he/she can simply go to the Settings and click on the option. Clicking on the options open up a dialog box with 3 input layouts for:

  1. Current Password
  2. New Password
  3. Confirm New Password

So, user can simply add these three inputs and click “Ok”. This will change the password of their account. Let’s see some code explanation.

  1. When user clicks on the “reset password” option from the settings, the showResetPasswordAlert() method is called which displays the dialog. And when user clicks on the “OK” button the resetPassword method() in the presenter is called passing input from the three input layout as parameters.

settingsPresenter.resetPassword(password.editText?.text.toString(), newPassword.editText?.text.toString(), conPassword.editText?.text.toString())

fun showResetPasswordAlert() {
   val builder = AlertDialog.Builder(activity)
   val resetPasswordView = activity.layoutInflater.inflate(R.layout.alert_reset_password, null)
   password = resetPasswordView.findViewById(R.id.password) as TextInputLayout
   newPassword = resetPasswordView.findViewById(R.id.newpassword) as TextInputLayout
   conPassword = resetPasswordView.findViewById(R.id.confirmpassword) as TextInputLayout
   builder.setView(resetPasswordView)
   builder.setTitle(Constant.CHANGE_PASSWORD)
           .setCancelable(false)
           .setNegativeButton(Constant.CANCEL, null)
           .setPositiveButton(getString(R.string.ok), null)
   resetPasswordAlert = builder.create()
   resetPasswordAlert.show()
   setupPasswordWatcher()
   resetPasswordAlert.getButton(AlertDialog.BUTTON_POSITIVE)?.setOnClickListener {
       settingsPresenter.resetPassword(password.editText?.text.toString(), newPassword.editText?.text.toString(), conPassword.editText?.text.toString())
   }
}
  1. In the resetPassword method, all details about the passwords are checked like:
  1. If passwords are not empty.
  2. If passwords’ lengths are greater than 6.
  3. If new password and confirmation new password matches

   

When all the conditions are satisfied and all the inputs are valid, resetPassword() in model is called which makes network call to change password of the user.

settingModel.resetPassword(password,newPassword,this)

override fun resetPassword(password: String, newPassword: String, conPassword: String) {
   if (password.isEmpty()) {
       settingView?.invalidCredentials(true, Constant.PASSWORD)
       return
   }
   if (newPassword.isEmpty()) {
       settingView?.invalidCredentials(true, Constant.NEW_PASSWORD)
       return
   }
   if (conPassword.isEmpty()) {
       settingView?.invalidCredentials(true, Constant.CONFIRM_PASSWORD)
       return
   }

   if (!CredentialHelper.isPasswordValid(newPassword)) {
       settingView?.passwordInvalid(Constant.NEW_PASSWORD)
       return
   }

   if (newPassword != conPassword) {
       settingView?.invalidCredentials(false, Constant.NEW_PASSWORD)
       return
   }
   settingModel.resetPassword(password,newPassword,this)
}

Summary

So, this blog talked about how the Change Password feature is implemented in SUSI Android App. This included how a network call is made, logic for making network, information about API, making dialogs with custom UI, etc. So, If you are looking forward to contribute to SUSI Android App, this can help you a little. But if not so, this may also help you in understanding and how you can implement a dialog box with custom UI.

References

  1. To know about servlets https://en.wikipedia.org/wiki/Java_servlet
  2. To see how to implement one https://www.javatpoint.com/servlet-tutorial
  3. To see how to make network calls in android using Retrofit https://guides.codepath.com/android/Consuming-APIs-with-Retrofit
  4. Official docs for displaying dialog https://developer.android.com/guide/topics/ui/dialogs.html
  5. Implementing dialog boxes with custom UI https://stackoverflow.com/questions/13341560/how-to-create-a-custom-dialog-box-in-android
  6. Pull Request for API reference: https://github.com/fossasia/susi_server/pull/352
Continue ReadingImplementing Change Password Feature in SUSI Android App using Custom Dialogs

Reactive Side Effects of Actions in Loklak Search

In a Redux based application, every component of the application is state driven. Redux based applications manage state in a predictable way, using a centralized Store, and Reducers to manipulate various aspects of the state. Each reducer controls a specific part of the state and this allows us to write the code which is testable, and state is shared between the components in a stable way, ie. there are no undesired mutations to the state from any components. This undesired mutation of the shared state is prevented by using a set of predefined functions called reducers which are central to the system and updates the state in a predictable way.

These reducers to update the state require some sort triggers to run. This blog post concentrates on these triggers, and how in turn these triggers get chained to form a Reactive Chaining of events which occur in a predictable way, and how this technique is used in latest application structure of Loklak Search. In any state based asynchronous application, like, Loklak Search the main issue with state management is to handle the asynchronous action streams in a predictable manner and to chain asynchronous events one after the other.  The technique of reactive action chaining solves the problem of dealing with asynchronous data streams in a predictable and manageable manner.

Overview

Actions are the triggers for the reducers, each redux action consists of a type and an optional payload. Type of the action is like its ID which should be purposely unique in the application. Each reducer function takes the current state which it controls and action which is dispatched. The reducer decides whether it needs to react to that action or not. If the user reacts to the action, it modifies the state according to the action payload and returns the modified state, else, it returns the original state. So at the core, the actions are like the triggers in the application, which make one or more reducers to work. This is the basic architecture of any redux application. The actions are the triggers and reducers are the state maintainers and modifiers. The only way to modify the state is via a reducer, and a reducer only runs when a corresponding action is dispatched.

Now, who dispatches these actions? This question is very important. The Actions can be technically dispatched from anywhere in the application, from components, from services, from directives, from pipes etc. But we almost in every situation will always want the action to be dispatched by the component. Component who wishes to modify the state dispatch the corresponding actions.

Reactive Effects

If the components are the one who dispatch the action, which triggers a reducer function which modifies the state, then what are these effects, cause the cycle of events seem pretty much complete. The Effects are the Side Effects, of a particular action. The term “side effect” means these are the piece of code which runs whenever an action is dispatched. Don’t confuse them with the reducer functions, effects are not same as the reducer functions as they are not associated with any state i.e. they don’t modify any state. They are the independent sources of other actions. What this means is whenever an Action is dispatched, and we want to dispatch some other action, maybe immediately or asynchronously, we use these side effects. So in a nutshell, effects are the piece of code which reacts to a particular action, and eventually dispatches some other actions.

The most common use case of effects is to call a corresponding service and fetch the data from the server, and then when the data is loaded, dispatch a SearchCompleteAction. These are the simplest of use cases of effects and are most commonly use in Loklak Search. This piece of code below shows how it is done.

@Effect()
search$: Observable<Action>
= this.actions$
.ofType(apiAction.ActionTypes.SEARCH)
.map((action: apiAction.SearchAction) => action.payload)
.switchMap(query => {
return this.apiSearchService.fetchQuery(query)
.map(response => new apiAction.SearchCompleteSuccessAction(response))

This is a very simple type of an effect, it filters out all the actions and react to only the type of action which we are interested in, here SEARCH, and then after calling the respective Service, it either dispatches SearchComplete or a SearchFail action depending on the status of the response from the API. The effect runs on SEARCH action and eventually dispatches, the success or the fail actions.

This scheme illustrates the effect as another point apart from components, to dispatch some action. The difference being, components dispatch action on the basis of the User inputs and events, whereas Effects dispatch actions on the basis of other actions.

Reactive Chaining of Actions

We can thus take the advantage of this approach in a form of reactive chaining of actions. This reactive chaining of actions means that some component dispatches some action, which as a side effects, dispatches some other action, and it dispatches another set of actions and so on. This means a single action dispatched from a component, brings about the series of actions which follow one another. This approach makes it possible to write reducers at the granular level rather than complete state level. As a series of actions can be set up which, start from a fine grain, and reaches out to a coarse grain. The loklak search application uses this technique to update the state of query. The reducers in the loklak search rather than updating the whole query structure update only the required part of the state. This helps in code maintainability as the one type of query attribute has no effect on the other type

@Effect()
inputChange$: Observable<Action>
= this.actions$
.ofType(queryAction.ActionTypes.VALUE_CHANGE)
.map(_ => new queryAction.QueryChangeAction(''));

@Effect()
filterChange$: Observable<Action>
= this.actions$
.ofType(queryAction.ActionTypes.FILTER_CHANGE)
.map(_ => new queryAction.QueryChangeAction(''));

Here the QUERY_CHANGE action further can do other processing of the query and then dispatch the SearchAction, which eventually calls the service and then return the response, then the success or fail actions can be dispatched eventually.

Conclusion

The reactive side effects is one of the most beautiful thing we can do with Redux and Reactive Programming. They provide an easy clean way to chain events in an application, which helps in a cleaner non-overlapping state management along with clean and simple reducers. This idea of the reactive chaining can be extended to any level of sophistication, and that too in a simple and easy to understand manner.

Resources and links

Continue ReadingReactive Side Effects of Actions in Loklak Search

Setting the Foreground Color Based on the Background Color in Phimpme Android

In the situations where the background color gets changed, the foreground color should also change to some other color. Otherwise the foreground item may not be properly visible on the background.

In Phimpme Android, we came across this situation when dealing with the bottom navigation bar. We have a preference option in our application which enables the user to change the primary color of the application. This change affects the background color of the bottom navigation bar also. We initially had white as selected foreground item’s color and grey as the unselected items’ color. It was fine for darker background colors. But problem arose when we changed the primary color of the application to the lighter shade of any color. The foreground item which was in white color was not clearly visible over the background. So we had to dynamically decide the foreground color when the background color gets changed.

Deciding the foreground color of the bottom navigation bar in Phimpme Android was tough. The foreground item should not only be visible on the background color but it should also be visibly attractive. We needed a minimal look. So we decided to use only two colors – black and white for the selected item’s color. The unselected items’ color can be fixed. A shade of gray with less alpha can be used for it.

The black and white colors are decided based on the lightness of the background color. We used the rgb value of the background color to find the lightness. We fixed a threshold for this lightness by visual judgement and selected the foreground color from the black and white colors based on this threshold lightness. If the lightness of the background color of the bottom navigation color of Phimpme Android application is less than the defined threshold value, we used white as the foreground color and vice-versa.

The lightness can be found out by taking the average of the red, green and blue values of the background color. We considered 100 as the threshold lightness for deciding the foreground color.

void setIconColor(int color){
   if(Color.red(color) + Color.green(color)+ Color.blue(color) < 300)
       colors[0] = Color.WHITE;
   else
       colors[0] = Color.BLACK;
}

For setting the different colors for different states of the foreground items on the bottom navigation bar, we used the setItemIconTintList method which takes ColorStatesList as an argument.

colors[1]  = ContextCompat.getColor(this, R.color.bottom_navigation_tabs);
ColorStateList bottomNavList = new ColorStateList(states, colors);
navigationView.setItemIconTintList(bottomNavList);

This above ColorStateList takes a two dimensional array of states for which we wanted to change the color of item and an array of corresponding colors for those states as arguments. In Phimpme Android we used the following way.

private int[][] states = new int[][] {
       new int[] {android.R.attr.state_checked}, // checked
       new int[] {-android.R.attr.state_checked}, // unchecked
};

private int[] colors = new int[] {
       Color.WHITE, // checked
       0 // unchecked set default in onCreate
};

Resources:

Continue ReadingSetting the Foreground Color Based on the Background Color in Phimpme Android

Configurable Settings for Repositories Registered to Yaydoc

Yaydoc, our automatic documentation generation and deployment project, generates and deploys documentation for each of its registered repositories. These repositories registered to Yaydoc have various configurable settings which can be edited to change the behavior of the build process and other processes surrounding it. These settings include

  • Report build status via email,
  • Get build status for pull requests to the repository,
  • Specify project branches,
  • Add or remove sub-projects to a registered master project, and
  • Enable or Disable the build process
  • Delete the repository

Report Build Status via Email

At Yaydoc, user has an option to receive a mail

  • On the first build after the repository is registered to Yaydoc, irrespective of the status
  • On every failed build
  • On the change of build status (Success to Failed or vice versa)
  • To the user who registered the repository to Yaydoc

It is possible that the user may register the repository from one email-address but wishes to receive the build status in another email. Furthermore, the user may never wish to receive any email from the repository. Keeping user’s experience at a priority, we also made it configurable for the user to disable the mail service.

This feature is implemented by adding a  mailService  attribute to the Repository Schema with status: Boolean and email: String as its sub-attributes. The feature is toggled, setting the value of ‘mailService.status’ to true or false.

const Repository = mongoose.model(‘Repository’, {
  ....
  ....
  mailService: {
    status: Boolean,
    email: String,
  },
  ....
  ....
});

Pull request implementing this feature: https://github.com/fossasia/yaydoc/pull/361/files

Get build status for pull requests to the repository

We made the generation of documentation website during each Pull Request to the registered repository. Since we do not want to increase the load on our server and also due to the fact that, since testing documentation generation during a Pull Request would be rare, and the user may not want to integrate Yaydoc to Pull Requests, we made this feature configurable with ‘disabled’ as the default configuration.

A detail description of the process can be found at: Showing Pull Request Build Status in Yaydoc | FOSSASIA blog

Specify Project Branches

The documentation for a registered repository is generated at every commit made to the repository. This process happens in commits made to any of the repository’s branches except for branches that they do not have a `.yaydoc.yml` file and the `gh-pages` branch. It is seen that In many of the open sources projects hosted on Github, the development flow follows an approach in which new features and patches are made in a separate branch of the same repository before sending a PR to merge it into the master repository. In such a case, generating documentation from all the branches is an overkill and would not be expected. Hence, we let the user specify branches from which the documentation should be generated and deployed.

The existing active branches of the repository along with the registered branches are retrieved and are used to display a Bootstrap select picker to specify branches.

router.get(‘/:owner/:repository/branches’, function (req, res, next) {
  var name = req.params.owner + ‘/’ + req.params.repository;
  async.parallel({
    branches: function (callback) {
      github.getRepositoryBranches(name, function (error, branches) {
        callback(error, branches);
      });
    },
    registeredBranches: function (callback) {
      var branches = [];
      for (var branches of registeredBranches) {
        branches.push(branch.name);
      }
      callback(error, branches);
    }
  }, function (error, results) {
    res.json({
      branches: results.branches,
      registeredBranches: results.registeredBranches
    });
  });
});

Once a user specify one or more of the existing branches of the repository, documentation build occurs only from those specified repositories if they contain the `.yaydoc.yml` file.

Pull request implementing this feature: https://github.com/fossasia/yaydoc/pull/424

Add or remove sub-projects to a registered master project

Apart from the generation of documentation of a single Github repository, we also offer the user to register sub-projects for a master project. The documentation generated from these subprojects are kept at a sub route of the website with auto-generated links at the start page. The addition and deletion of these sub-projects is also made configurable from repository settings.

Pull request implementing this feature: https://github.com/fossasia/yaydoc/pull/318

Resources:

  1. Async utilities for node and browser – https://caolan.github.io/async/
  2. Mongoose Schema Documentation: http://mongoosejs.com/docs/guide.html
Continue ReadingConfigurable Settings for Repositories Registered to Yaydoc

Deleting SUSI Skills from Server

SUSI Skill CMS is a web application to create and edit skills. In this blog post I will be covering how we made the skill deleting feature in Skill CMS from the SUSI Server.
The deletion of skill was to be made in such a way that user can click a button to delete the skill. As soon as they click the delete button the skill is deleted it is removed from the directory of SUSI Skills. But admins have an option to recover the deleted skill before completion of 30 days of deleting the skill.

First we will accept all the request parameters from the GET request.

        String model_name = call.get("model", "general");
        String group_name = call.get("group", "Knowledge");
        String language_name = call.get("language", "en");
        String skill_name = call.get("skill", "wikipedia");

In this we get the model name, category, language name, skill name and the commit ID. The above 4 parameters are used to make a file path that is used to find the location of the skill in the Susi Skill Data repository.

 if(!DAO.deleted_skill_dir.exists()){
            DAO.deleted_skill_dir.mkdirs();
   }

We need to move the skill to a directory called deleted_skills_dir. So we check if the directory exists or not. If it not exists then we create a directory for the deleted skills.

  if (skill.exists()) {
   File file = new File(DAO.deleted_skill_dir.getPath()+path);
   file.getParentFile().mkdirs();
   if(skill.renameTo(file)){
   Boolean changed =  new File(DAO.deleted_skill_dir.getPath()+path).setLastModified(System.currentTimeMillis());
     }

This is the part where the real deletion happens. We get the path of the skill and rename that to a new path which is in the directory of deleted skills.

Also here change the last modified time of the skill as the current time. This time is used to check if the skill deleted is older than 30 days or not.

    try (Git git = DAO.getGit()) {
                DAO.pushCommit(git, "Deleted " + skill_name, rights.getIdentity().isEmail() ? rights.getIdentity().getName() : "anonymous@");
                json.put("accepted", true);
                json.put("message", "Deleted " + skill_name);
            } catch (IOException | GitAPIException e) {

Finally we add the changes to Git. DAO.pushCommit pushes to commit to the Susi Skill Data repository. If the user is logged in we get the email of the user and set that email as the commit author. Else we set the username “anonymous@”.

Then in the caretaker class there is a method deleteOldFiles that checks for all the files whose last modified time was older than 30 days. If there is any file whose last modified time was older than 30 days then it quietly delete the files.

public void deleteOldFiles() {
     Collection<File> filesToDelete = FileUtils.listFiles(new         File(DAO.deleted_skill_dir.getPath()),
     new 
(DateTime.now().withTimeAtStartOfDay().minusDays(30).toDate()),
            TrueFileFilter.TRUE);    // include sub dirs
        for (File file : filesToDelete) {
               boolean success = FileUtils.deleteQuietly(file);
            if (!success) {
                System.out.print("Deleted skill older than 30 days.");
            }
      }
}

To test this API endpoint, we need to call http://localhost:4000/cms/deleteSkill.txt?model=general&group=Knowledge&language=en&skill=<skill_name>

Resources

JGit Documentation: https://eclipse.org/jgit/documentation/

Commons IO: https://commons.apache.org/proper/commons-io/

Age Filter: https://commons.apache.org/proper/commons-io/javadocs/api-1.4/org/apache/commons/io/filefilter/AgeFileFilter.html

JGit User Guide: http://wiki.eclipse.org/JGit/User_Guide

JGit Repository access: http://www.codeaffine.com/2014/09/22/access-git-repository-with-jgit/

Continue ReadingDeleting SUSI Skills from Server

Getting SUSI Skill at a Commit ID

Susi Skill CMS is a web app to edit and create new skills. We use Git for storing different versions of Susi Skills. So what if we want to roll back to a previous version of the skill? To implement this feature in Susi Skill CMS, we needed an API endpoint which accepts the name of the skill and the commit ID and returns the file at that commit ID.

In this blog post I will tell about making an API endpoint which works similar to git show.

First we will accept all the request parameters from the GET request.

        String model_name = call.get("model", "general");
        String group_name = call.get("group", "Knowledge");
        String language_name = call.get("language", "en");
        String skill_name = call.get("skill", "wikipedia");
        String commitID  = call.get("commitID", null);

In this we get the model name, category, language name, skill name and the commit ID. The above 4 parameters are used to make a file path that is used to find the location of the skill in the Susi Skill Data repository.

This servlet need CommitID to work and if commit ID is not given in the request parameters then we send an error message saying that the commit id is null and stop the servlet execution.

    Repository repository = DAO.getRepository();
    ObjectId CommitIdObject = repository.resolve(commitID);

Then we get the git repository of the skill from the DAO and initialize the repository object.

From the commitID that we got in the request parameters we create a CommitIdObject.

   (RevWalk revWalk = new RevWalk(repository)) {
   RevCommit commit = revWalk.parseCommit(CommitIdObject);
   RevTree tree = commit.getTree();


Now using commit’s tree, we will find the find the path and get the tree of the commit.

From the TreeWalk in the repository we will set a filter to find a file. This searches recursively for the files inside all the folders.

                revWalk = new RevWalk(repository)) {
                try (TreeWalk treeWalk = new TreeWalk(repository)) {
                    treeWalk.addTree(tree);
                    treeWalk.setRecursive(true);
                    treeWalk.setFilter(PathFilter.create(path));
                    if (!treeWalk.next()) {
                        throw new IllegalStateException("Did not find expected file");
                    }

If the TreeWalk reaches to an end and does not find the specified skill path then it returns anIllegal State Exception with an message saying did not found the file on that commit ID.

       ObjectId objectId = treeWalk.getObjectId(0);
       ObjectLoader loader = repository.open(objectId);
       OutputStream output = new OutputStream();
       loader.copyTo(output);

And then one can the loader to read the file. From the treeWalk we get the object and create an output stream to copy the file content in it. After that we create the JSON and put the OutputStream object as as String in it.

       json.put("file",output);

This Servlet can be seen working api.susi.ai: http://api.susi.ai/cms/getFileAtCommitID.json?model=general&group=knowledge&language=en&skill=bitcoin&commitID=214791f55c19f24d7744364495541b685539a4ee

Resources

JGit Documentation: https://eclipse.org/jgit/documentation/

JGit User Guide: http://wiki.eclipse.org/JGit/User_Guide

JGit Repository access: http://www.codeaffine.com/2014/09/22/access-git-repository-with-jgit/

JGit Github: https://github.com/eclipse/jgit

Continue ReadingGetting SUSI Skill at a Commit ID

Handling Errors While Parsing the yaml File in Yaydoc

Yaydoc, our automatic documentation generator uses a yaml file to read a user’s configuration. The internal configuration parser basically converts the yaml file to a python dictionary. Then, it serializes the values of that dictionary using a custom serialization format. From there it associates those values with environment variables which are then passed to bash scripts for various tasks such as deployment, generation, etc.. Some of those environment variables are again passed to another python layer which interacts with sphinx where they are deserialized before use. This whole system works pretty well for our use cases.

Now let’s assume a user adds a yaml file where they have a malformed section in the file. For example, to specify a theme, one needs to add the following to the yaml file.

build:
  theme:
    name: sphinx_fossasia_theme

But our user has the following in their yaml file.

build:
  theme: sphinx_fossasia_theme

Now this will raise an error as we expect a dictionary as a value for the key ‘theme’ but we got a string. Now how do we handle such cases without ignoring the entire file as that would be too much of a penalty for such a small mistake? One approach would have been to wrap each call to connect with a bunch of try-catch but that would render the code unreadable as the initial motivation for implementing the connect method was to abstract the internal implementation so that other contributors who may not be well versed with python can also easily add config options without needing to learn a bunch of python constructs.

So, what we did was that, while merging the dictionary containing default options and the dictionary containing the user preferences, we check whether the default has the same data type as that of the incoming value. If they are, It’s deemed safe to merge. There are certain relaxations though, like if the current type is a list, then the incoming value can be of any time as that can always be converted to a list of a single element. This is required to support the following syntax.

key:
  - value
key: value

The above two blocks are equivalent due to the above-mentioned approach although the type is different.

Now, after this pre-validation step is over we can ensure that the if the assumed type for a key is let’s say a dictionary, then it would be a dictionary. Hence no type errors would be raised like trying to access a dict method for another object, say a string which happened with the earlier implementation. After this, an extra parameter was added to the connect method to which we can now pass a validation function which if returns false, those values would be ignored. Usage of this feature has been implemented to a small level where we validate the links to subprojects and if they look like a valid github repo only then will they be included. Note that their existence is not checked. Only a regex based validation is performed.

It was also important to notify the user about these events when we detect that a specific section is invalid and provide informative and helpful error messages without failing the build. Hence proper error messages were also added which were informative so that the user knows exactly which section is to blame. This is similar to compilers where the error message is crucial to debug a certain piece of code.

Resources

Continue ReadingHandling Errors While Parsing the yaml File in Yaydoc