Shift from Java to Kotlin in SUSI Android

Previously SUSI Android was written in JAVA. But recently Google announced that it will officially support Kotlin on Android as a first class language so we decided to shift from Java to Kotlin. Kotlin runs on Java Virtual Machine and hence we can use Kotlin for Android app development. Kotlin is new programming language developed by JetBrains, which also developed IntelliJ-the IDE that Android Studio is based on. So Android Studio has excellent support for Kotlin.

Advantages of Kotlin over Java

  • Kotlin is a null safe language. It changes all the instances used in the code to non nullable type thus it ensures that the developers don’t get any nullPointerException.
  • Good Android Studio support. Kotlin tool is included with Android Studio 3.0 by default. But for Android Studio 2.x we need to install the required plugin for Kotlin.
  • Kotlin also provides support for Lambda function and other high order functions.
  • One of Kotlin’s greatest strengths as a potential alternative to Java is interoperability between Java and Kotlin. You can even have Java and Kotlin code existing side by side in the same project.
  • Kotlin provides the way to declare Extension function similar to that of C# and Gosu. We can use this function in the same way as we use the member functions of our class.

After seeing the above points it is now clear that Kotlin is much more effective than Java. In this blog post, I will show you how ForgotPasswordActivity is shifted from JAVA to Kotlin in SUSI Android.

How to install Kotlin plugin in Android Studio 2.x

We are using Android Studio 2.3.0 in our android project. To use Kotlin in Android Studio 2.x we need to install Kotlin plugin.

Steps to include Kotlin plugin in Android Studio 2.x are:

  • Go to File
  • After that select Settings
  • Then select Plugin Option from the left sidebar.
  • After that click on Browse repositories.
  • Search for Kotlin and install it.

Once it is installed successfully, restart Android Studio. Now configure Kotlin in Android Studio.

Shift  SUSI Android from JAVA to Kotlin

Kotlin is compatible with JAVA. Thus it allows us to change the code bit by bit instead of all at once. In SUSI Android app we implemented MVP architecture with Kotlin. We converted the code by one activity each time from JAVA to Kotlin. I converted the ForgotPassword of SUSI Android from JAVA to Kotlin with MVP architecture. I will discuss only shifting of SUSI Android from JAVA to Kotlin and not about MVP architecture implementation.

First thing is how to extend parent class. In JAVA we need to use extend keyword

public class ForgotPasswordActivity extends AppCompatActivity

but in Kotlin parent class is extended like

class ForgotPasswordActivity : AppCompatActivity ()

Second is no need to bind view in Kotlin. In JAVA we bind view using Butter Knife.

@BindView(R.id.forgot_email)

protected TextInputLayout email;

and

email.setError(getString(R.string.email_invalid_title));

but in Kotlin we can directly use view using id.

forgot_email.error = getString(R.string.email_invalid_title)

Another important thing is that instead of using setError and getError we can use only error for both purposes.

forgot_email.error

It can be used to get error in TextInputLayout ‘forgot_email’ and

forgot_email.error = getString(R.string.email_invalid_title)

can be use to set error in TextInputLayout ‘forgot_email’.

Third one is in Kotlin we don’t need to define variable type always

val notSuccessAlertboxHelper = AlertboxHelper(this@ForgotPasswordActivity, title, message, null, null, button, null, color)

but in JAVA we need to define it always.

AlertboxHelper notSuccessAlertboxHelper = new AlertboxHelper(ForgotPasswordActivity.this, title, message, null, null, positiveButtonText, null, colour);

And one of the most important feature of Kotlin null safety. So you don’t need to worry about NullPointerException. In java we need to check for null otherwise our app crash

if (url.getEditText() == null)

      throw new IllegalArgumentException(“No Edittext hosted!”);

But in Kotlin there is no need to check for null. You need to use ’?’ and Kotlin will handle it itself.  

input_url.editText?.text.toString()

You can find the previous ForgotPasswordActivity here. You can compare it with new ForgotPaswordActivity.

Reference

Continue ReadingShift from Java to Kotlin in SUSI Android

Implementing Change Password Feature in SUSI Android App using Custom Dialogs

Recently a new servlet was implemented on the SUSI Server about changing the password of the logged in user. This feature comes in handy to avoid unauthorized usage of the SUSI Account. Almost all the online platforms have this feature to change the password to avoid notorious user to unethical use someone else’s account. In SUSI Android app this new API was used with a nice UI to change the password of the user. The process is very simple and easy to grasp. This blog will try to cover the API information and implementation of the Change Password feature in the android client.

API Information

For changing the password of SUSI Account of the user, we have to call on  /aaa/changepassword.json

We have to provide three parameters along with this api call:

  1. changepassword:  Email of user (type string) using which user is logged in.
  2. password:  Old password (type string with min length of 6) of the user.
  3. newpassword: New password (type string with min length of 6) of the user.
  4. access_token: An encrypted access_token indicating user is logged in.

Sample Response (Success)

{
  "session": {"identity": {
    "type": "email",
    "name": "YOUR_EMAIL_ADDRESS",
    "anonymous": false
  }},
  "accepted": true,
  "message": "Your password has been changed!"
}

Error Response (Failure). This happens when user is not logged in:

HTTP ERROR 401
Problem accessing /aaa/changepassword.json. Reason:
   Base user role not sufficient. Your base user role is 'ANONYMOUS', your user role is 'anonymous'

Implementation in SUSI Android App

The change password option is located in Settings Activity and displayed only when user is logged in. So, if a logged in user wants to change the password of his/her SUSI AI account, he/she can simply go to the Settings and click on the option. Clicking on the options open up a dialog box with 3 input layouts for:

  1. Current Password
  2. New Password
  3. Confirm New Password

So, user can simply add these three inputs and click “Ok”. This will change the password of their account. Let’s see some code explanation.

  1. When user clicks on the “reset password” option from the settings, the showResetPasswordAlert() method is called which displays the dialog. And when user clicks on the “OK” button the resetPassword method() in the presenter is called passing input from the three input layout as parameters.

settingsPresenter.resetPassword(password.editText?.text.toString(), newPassword.editText?.text.toString(), conPassword.editText?.text.toString())

fun showResetPasswordAlert() {
   val builder = AlertDialog.Builder(activity)
   val resetPasswordView = activity.layoutInflater.inflate(R.layout.alert_reset_password, null)
   password = resetPasswordView.findViewById(R.id.password) as TextInputLayout
   newPassword = resetPasswordView.findViewById(R.id.newpassword) as TextInputLayout
   conPassword = resetPasswordView.findViewById(R.id.confirmpassword) as TextInputLayout
   builder.setView(resetPasswordView)
   builder.setTitle(Constant.CHANGE_PASSWORD)
           .setCancelable(false)
           .setNegativeButton(Constant.CANCEL, null)
           .setPositiveButton(getString(R.string.ok), null)
   resetPasswordAlert = builder.create()
   resetPasswordAlert.show()
   setupPasswordWatcher()
   resetPasswordAlert.getButton(AlertDialog.BUTTON_POSITIVE)?.setOnClickListener {
       settingsPresenter.resetPassword(password.editText?.text.toString(), newPassword.editText?.text.toString(), conPassword.editText?.text.toString())
   }
}
  1. In the resetPassword method, all details about the passwords are checked like:
  1. If passwords are not empty.
  2. If passwords’ lengths are greater than 6.
  3. If new password and confirmation new password matches

   

When all the conditions are satisfied and all the inputs are valid, resetPassword() in model is called which makes network call to change password of the user.

settingModel.resetPassword(password,newPassword,this)

override fun resetPassword(password: String, newPassword: String, conPassword: String) {
   if (password.isEmpty()) {
       settingView?.invalidCredentials(true, Constant.PASSWORD)
       return
   }
   if (newPassword.isEmpty()) {
       settingView?.invalidCredentials(true, Constant.NEW_PASSWORD)
       return
   }
   if (conPassword.isEmpty()) {
       settingView?.invalidCredentials(true, Constant.CONFIRM_PASSWORD)
       return
   }

   if (!CredentialHelper.isPasswordValid(newPassword)) {
       settingView?.passwordInvalid(Constant.NEW_PASSWORD)
       return
   }

   if (newPassword != conPassword) {
       settingView?.invalidCredentials(false, Constant.NEW_PASSWORD)
       return
   }
   settingModel.resetPassword(password,newPassword,this)
}

Summary

So, this blog talked about how the Change Password feature is implemented in SUSI Android App. This included how a network call is made, logic for making network, information about API, making dialogs with custom UI, etc. So, If you are looking forward to contribute to SUSI Android App, this can help you a little. But if not so, this may also help you in understanding and how you can implement a dialog box with custom UI.

References

  1. To know about servlets https://en.wikipedia.org/wiki/Java_servlet
  2. To see how to implement one https://www.javatpoint.com/servlet-tutorial
  3. To see how to make network calls in android using Retrofit https://guides.codepath.com/android/Consuming-APIs-with-Retrofit
  4. Official docs for displaying dialog https://developer.android.com/guide/topics/ui/dialogs.html
  5. Implementing dialog boxes with custom UI https://stackoverflow.com/questions/13341560/how-to-create-a-custom-dialog-box-in-android
  6. Pull Request for API reference: https://github.com/fossasia/susi_server/pull/352
Continue ReadingImplementing Change Password Feature in SUSI Android App using Custom Dialogs

Reactive Side Effects of Actions in Loklak Search

In a Redux based application, every component of the application is state driven. Redux based applications manage state in a predictable way, using a centralized Store, and Reducers to manipulate various aspects of the state. Each reducer controls a specific part of the state and this allows us to write the code which is testable, and state is shared between the components in a stable way, ie. there are no undesired mutations to the state from any components. This undesired mutation of the shared state is prevented by using a set of predefined functions called reducers which are central to the system and updates the state in a predictable way.

These reducers to update the state require some sort triggers to run. This blog post concentrates on these triggers, and how in turn these triggers get chained to form a Reactive Chaining of events which occur in a predictable way, and how this technique is used in latest application structure of Loklak Search. In any state based asynchronous application, like, Loklak Search the main issue with state management is to handle the asynchronous action streams in a predictable manner and to chain asynchronous events one after the other.  The technique of reactive action chaining solves the problem of dealing with asynchronous data streams in a predictable and manageable manner.

Overview

Actions are the triggers for the reducers, each redux action consists of a type and an optional payload. Type of the action is like its ID which should be purposely unique in the application. Each reducer function takes the current state which it controls and action which is dispatched. The reducer decides whether it needs to react to that action or not. If the user reacts to the action, it modifies the state according to the action payload and returns the modified state, else, it returns the original state. So at the core, the actions are like the triggers in the application, which make one or more reducers to work. This is the basic architecture of any redux application. The actions are the triggers and reducers are the state maintainers and modifiers. The only way to modify the state is via a reducer, and a reducer only runs when a corresponding action is dispatched.

Now, who dispatches these actions? This question is very important. The Actions can be technically dispatched from anywhere in the application, from components, from services, from directives, from pipes etc. But we almost in every situation will always want the action to be dispatched by the component. Component who wishes to modify the state dispatch the corresponding actions.

Reactive Effects

If the components are the one who dispatch the action, which triggers a reducer function which modifies the state, then what are these effects, cause the cycle of events seem pretty much complete. The Effects are the Side Effects, of a particular action. The term “side effect” means these are the piece of code which runs whenever an action is dispatched. Don’t confuse them with the reducer functions, effects are not same as the reducer functions as they are not associated with any state i.e. they don’t modify any state. They are the independent sources of other actions. What this means is whenever an Action is dispatched, and we want to dispatch some other action, maybe immediately or asynchronously, we use these side effects. So in a nutshell, effects are the piece of code which reacts to a particular action, and eventually dispatches some other actions.

The most common use case of effects is to call a corresponding service and fetch the data from the server, and then when the data is loaded, dispatch a SearchCompleteAction. These are the simplest of use cases of effects and are most commonly use in Loklak Search. This piece of code below shows how it is done.

@Effect()
search$: Observable<Action>
= this.actions$
.ofType(apiAction.ActionTypes.SEARCH)
.map((action: apiAction.SearchAction) => action.payload)
.switchMap(query => {
return this.apiSearchService.fetchQuery(query)
.map(response => new apiAction.SearchCompleteSuccessAction(response))

This is a very simple type of an effect, it filters out all the actions and react to only the type of action which we are interested in, here SEARCH, and then after calling the respective Service, it either dispatches SearchComplete or a SearchFail action depending on the status of the response from the API. The effect runs on SEARCH action and eventually dispatches, the success or the fail actions.

This scheme illustrates the effect as another point apart from components, to dispatch some action. The difference being, components dispatch action on the basis of the User inputs and events, whereas Effects dispatch actions on the basis of other actions.

Reactive Chaining of Actions

We can thus take the advantage of this approach in a form of reactive chaining of actions. This reactive chaining of actions means that some component dispatches some action, which as a side effects, dispatches some other action, and it dispatches another set of actions and so on. This means a single action dispatched from a component, brings about the series of actions which follow one another. This approach makes it possible to write reducers at the granular level rather than complete state level. As a series of actions can be set up which, start from a fine grain, and reaches out to a coarse grain. The loklak search application uses this technique to update the state of query. The reducers in the loklak search rather than updating the whole query structure update only the required part of the state. This helps in code maintainability as the one type of query attribute has no effect on the other type

@Effect()
inputChange$: Observable<Action>
= this.actions$
.ofType(queryAction.ActionTypes.VALUE_CHANGE)
.map(_ => new queryAction.QueryChangeAction(''));

@Effect()
filterChange$: Observable<Action>
= this.actions$
.ofType(queryAction.ActionTypes.FILTER_CHANGE)
.map(_ => new queryAction.QueryChangeAction(''));

Here the QUERY_CHANGE action further can do other processing of the query and then dispatch the SearchAction, which eventually calls the service and then return the response, then the success or fail actions can be dispatched eventually.

Conclusion

The reactive side effects is one of the most beautiful thing we can do with Redux and Reactive Programming. They provide an easy clean way to chain events in an application, which helps in a cleaner non-overlapping state management along with clean and simple reducers. This idea of the reactive chaining can be extended to any level of sophistication, and that too in a simple and easy to understand manner.

Resources and links

Continue ReadingReactive Side Effects of Actions in Loklak Search

Full-Screen Speech Input Implementation in SUSI Android App

SUSI Android has some very good features and one of them is, it can take input in speech format from user i.e if the user says anything then it can detect it and convert it to text. This feature is implemented in SUSI Android app using Android’s built-in speech-to-text functionality. You can implement Android’s built-in speech-to-text functionality using either only RecognizerIntent class or SpeechRecognizerIntent class, RecognitionListner interface and RecognizerIntent class. Using former method has some disadvantages:

  • During speech input, it shows a dialog box (as shown here) and it breaks the connection between user and app.
  • We can’t show partial result i.e text to the user but using the later method we can show it.

We used  SpeechRecognizerIntent class, RecognitionListner interface and RecognizerIntent class to implement Android’s built-in speech-to-text functionality in SUSI Android and you know the reason for that. In this blog post, I will show you how I implemented this feature in SUSI Android with new UI.

Layout design

You can give speech input to SUSI Android either by clicking mic button

or using ‘Hi SUSI’ hotword. When you click on mic button or use ‘Hi SUSI’ hotword, you can see a screen where you will give speech input.

Two important part of this layout are:

<TextView

  android:id=“@+id/txtchat”

  android:layout_width=“wrap_content”

  android:layout_height=“wrap_content”

  

 />

  • TextView: It used to show the partial result of speech input i.e it will show converted text (partial) of your speech.
<org.fossasia.susi.ai.speechinputanimation.SpeechProgressView

  android:id=“@+id/speechprogress”

  android:layout_width=“match_parent”

  android:layout_height=“50dp”

  android:layout_margin=“8dp”

  android:layout_gravity=“center”/>

  • SpeechProgressView: It is a custom view which use to show the animation when the user gives speech input. When the user starts speaking, the animation starts. This custom view contains five bars and these five bars animate according to user input.

Full-screen speech input implementation

When the user clicks on mic button or uses ‘Hi SUSI’ hotword, a screen comes where the user can give speech input. As already mentioned I used SpeechRecognizerIntent class, RecognitionListner interface and RecognizerIntent class to implement speech-to-text functionality in SUSI Android. RecognizerIntent class starts an intent and asks for speech input

val intent = Intent(RecognizerIntent.ACTION_RECOGNIZE_SPEECH)

intent.putExtra(RecognizerIntent.EXTRA_LANGUAGE_MODEL,

    RecognizerIntent.LANGUAGE_MODEL_FREE_FORM)

intent.putExtra(RecognizerIntent.EXTRA_CALLING_PACKAGE, “com.domain.app”)

intent.putExtra(RecognizerIntent.EXTRA_PARTIAL_RESULTS, true)

and send it through the speech recognizer. It does it through ACTION_RECOGNIZE_SPEECH. SpeechRecognizer class provides access to the speech recognition service. This service allows access to the speech recognizer and recognition related event occurs RecognitionListner receive notification from SpeechRecognizer class.

recognizer = SpeechRecognizer

      .createSpeechRecognizer(activity.applicationContext)

val listener = object : RecognitionListener {

 //implement all override methods

}

When the user starts speaking, the height of bars changes according to change in sound level. When sound level changes, onRmsChanged method get called where we are calling onRmsChanged method of SpeechProgressView class which is responsible for animating bars according to change in sound level.

override fun onRmsChanged(rmsdB: Float) {

  if (speechprogress != null)

      speechprogress.onRmsChanged(rmsdB)

}

When user finished speaking onEndOfSpeech method get called where we call onEndOfSpeech method of SpeechProgressView class which is responsible for rotating animation. Rotation is used to show that SUSI Android has finished listening and now it is processing your input.

override fun onEndOfSpeech() {

  if (speechprogress != null)

      speechprogress.onEndOfSpeech()

}

In case of any error, onError method get called and in case of successful speech input, onResults method get called. In both cases, we reset bars to their initial position and show chat activity user. The user can again give speech input either by clicking on mic or using ‘Hi SUSI’ hotword.

override fun onResults(results: Bundle) {

  if (speechprogress != null)

      speechprogress.onResultOrOnError()

  activity.supportFragmentManager.popBackStackImmediate()

}

Reference

Continue ReadingFull-Screen Speech Input Implementation in SUSI Android App

Configurable Settings for Repositories Registered to Yaydoc

Yaydoc, our automatic documentation generation and deployment project, generates and deploys documentation for each of its registered repositories. These repositories registered to Yaydoc have various configurable settings which can be edited to change the behavior of the build process and other processes surrounding it. These settings include

  • Report build status via email,
  • Get build status for pull requests to the repository,
  • Specify project branches,
  • Add or remove sub-projects to a registered master project, and
  • Enable or Disable the build process
  • Delete the repository

Report Build Status via Email

At Yaydoc, user has an option to receive a mail

  • On the first build after the repository is registered to Yaydoc, irrespective of the status
  • On every failed build
  • On the change of build status (Success to Failed or vice versa)
  • To the user who registered the repository to Yaydoc

It is possible that the user may register the repository from one email-address but wishes to receive the build status in another email. Furthermore, the user may never wish to receive any email from the repository. Keeping user’s experience at a priority, we also made it configurable for the user to disable the mail service.

This feature is implemented by adding a  mailService  attribute to the Repository Schema with status: Boolean and email: String as its sub-attributes. The feature is toggled, setting the value of ‘mailService.status’ to true or false.

const Repository = mongoose.model(‘Repository’, {
  ....
  ....
  mailService: {
    status: Boolean,
    email: String,
  },
  ....
  ....
});

Pull request implementing this feature: https://github.com/fossasia/yaydoc/pull/361/files

Get build status for pull requests to the repository

We made the generation of documentation website during each Pull Request to the registered repository. Since we do not want to increase the load on our server and also due to the fact that, since testing documentation generation during a Pull Request would be rare, and the user may not want to integrate Yaydoc to Pull Requests, we made this feature configurable with ‘disabled’ as the default configuration.

A detail description of the process can be found at: Showing Pull Request Build Status in Yaydoc | FOSSASIA blog

Specify Project Branches

The documentation for a registered repository is generated at every commit made to the repository. This process happens in commits made to any of the repository’s branches except for branches that they do not have a `.yaydoc.yml` file and the `gh-pages` branch. It is seen that In many of the open sources projects hosted on Github, the development flow follows an approach in which new features and patches are made in a separate branch of the same repository before sending a PR to merge it into the master repository. In such a case, generating documentation from all the branches is an overkill and would not be expected. Hence, we let the user specify branches from which the documentation should be generated and deployed.

The existing active branches of the repository along with the registered branches are retrieved and are used to display a Bootstrap select picker to specify branches.

router.get(‘/:owner/:repository/branches’, function (req, res, next) {
  var name = req.params.owner + ‘/’ + req.params.repository;
  async.parallel({
    branches: function (callback) {
      github.getRepositoryBranches(name, function (error, branches) {
        callback(error, branches);
      });
    },
    registeredBranches: function (callback) {
      var branches = [];
      for (var branches of registeredBranches) {
        branches.push(branch.name);
      }
      callback(error, branches);
    }
  }, function (error, results) {
    res.json({
      branches: results.branches,
      registeredBranches: results.registeredBranches
    });
  });
});

Once a user specify one or more of the existing branches of the repository, documentation build occurs only from those specified repositories if they contain the `.yaydoc.yml` file.

Pull request implementing this feature: https://github.com/fossasia/yaydoc/pull/424

Add or remove sub-projects to a registered master project

Apart from the generation of documentation of a single Github repository, we also offer the user to register sub-projects for a master project. The documentation generated from these subprojects are kept at a sub route of the website with auto-generated links at the start page. The addition and deletion of these sub-projects is also made configurable from repository settings.

Pull request implementing this feature: https://github.com/fossasia/yaydoc/pull/318

Resources:

  1. Async utilities for node and browser – https://caolan.github.io/async/
  2. Mongoose Schema Documentation: http://mongoosejs.com/docs/guide.html
Continue ReadingConfigurable Settings for Repositories Registered to Yaydoc

Deleting SUSI Skills from Server

SUSI Skill CMS is a web application to create and edit skills. In this blog post I will be covering how we made the skill deleting feature in Skill CMS from the SUSI Server.
The deletion of skill was to be made in such a way that user can click a button to delete the skill. As soon as they click the delete button the skill is deleted it is removed from the directory of SUSI Skills. But admins have an option to recover the deleted skill before completion of 30 days of deleting the skill.

First we will accept all the request parameters from the GET request.

        String model_name = call.get("model", "general");
        String group_name = call.get("group", "Knowledge");
        String language_name = call.get("language", "en");
        String skill_name = call.get("skill", "wikipedia");

In this we get the model name, category, language name, skill name and the commit ID. The above 4 parameters are used to make a file path that is used to find the location of the skill in the Susi Skill Data repository.

 if(!DAO.deleted_skill_dir.exists()){
            DAO.deleted_skill_dir.mkdirs();
   }

We need to move the skill to a directory called deleted_skills_dir. So we check if the directory exists or not. If it not exists then we create a directory for the deleted skills.

  if (skill.exists()) {
   File file = new File(DAO.deleted_skill_dir.getPath()+path);
   file.getParentFile().mkdirs();
   if(skill.renameTo(file)){
   Boolean changed =  new File(DAO.deleted_skill_dir.getPath()+path).setLastModified(System.currentTimeMillis());
     }

This is the part where the real deletion happens. We get the path of the skill and rename that to a new path which is in the directory of deleted skills.

Also here change the last modified time of the skill as the current time. This time is used to check if the skill deleted is older than 30 days or not.

    try (Git git = DAO.getGit()) {
                DAO.pushCommit(git, "Deleted " + skill_name, rights.getIdentity().isEmail() ? rights.getIdentity().getName() : "anonymous@");
                json.put("accepted", true);
                json.put("message", "Deleted " + skill_name);
            } catch (IOException | GitAPIException e) {

Finally we add the changes to Git. DAO.pushCommit pushes to commit to the Susi Skill Data repository. If the user is logged in we get the email of the user and set that email as the commit author. Else we set the username “anonymous@”.

Then in the caretaker class there is a method deleteOldFiles that checks for all the files whose last modified time was older than 30 days. If there is any file whose last modified time was older than 30 days then it quietly delete the files.

public void deleteOldFiles() {
     Collection<File> filesToDelete = FileUtils.listFiles(new         File(DAO.deleted_skill_dir.getPath()),
     new 
(DateTime.now().withTimeAtStartOfDay().minusDays(30).toDate()),
            TrueFileFilter.TRUE);    // include sub dirs
        for (File file : filesToDelete) {
               boolean success = FileUtils.deleteQuietly(file);
            if (!success) {
                System.out.print("Deleted skill older than 30 days.");
            }
      }
}

To test this API endpoint, we need to call http://localhost:4000/cms/deleteSkill.txt?model=general&group=Knowledge&language=en&skill=<skill_name>

Resources

JGit Documentation: https://eclipse.org/jgit/documentation/

Commons IO: https://commons.apache.org/proper/commons-io/

Age Filter: https://commons.apache.org/proper/commons-io/javadocs/api-1.4/org/apache/commons/io/filefilter/AgeFileFilter.html

JGit User Guide: http://wiki.eclipse.org/JGit/User_Guide

JGit Repository access: http://www.codeaffine.com/2014/09/22/access-git-repository-with-jgit/

Continue ReadingDeleting SUSI Skills from Server

Getting SUSI Skill at a Commit ID

Susi Skill CMS is a web app to edit and create new skills. We use Git for storing different versions of Susi Skills. So what if we want to roll back to a previous version of the skill? To implement this feature in Susi Skill CMS, we needed an API endpoint which accepts the name of the skill and the commit ID and returns the file at that commit ID.

In this blog post I will tell about making an API endpoint which works similar to git show.

First we will accept all the request parameters from the GET request.

        String model_name = call.get("model", "general");
        String group_name = call.get("group", "Knowledge");
        String language_name = call.get("language", "en");
        String skill_name = call.get("skill", "wikipedia");
        String commitID  = call.get("commitID", null);

In this we get the model name, category, language name, skill name and the commit ID. The above 4 parameters are used to make a file path that is used to find the location of the skill in the Susi Skill Data repository.

This servlet need CommitID to work and if commit ID is not given in the request parameters then we send an error message saying that the commit id is null and stop the servlet execution.

    Repository repository = DAO.getRepository();
    ObjectId CommitIdObject = repository.resolve(commitID);

Then we get the git repository of the skill from the DAO and initialize the repository object.

From the commitID that we got in the request parameters we create a CommitIdObject.

   (RevWalk revWalk = new RevWalk(repository)) {
   RevCommit commit = revWalk.parseCommit(CommitIdObject);
   RevTree tree = commit.getTree();


Now using commit’s tree, we will find the find the path and get the tree of the commit.

From the TreeWalk in the repository we will set a filter to find a file. This searches recursively for the files inside all the folders.

                revWalk = new RevWalk(repository)) {
                try (TreeWalk treeWalk = new TreeWalk(repository)) {
                    treeWalk.addTree(tree);
                    treeWalk.setRecursive(true);
                    treeWalk.setFilter(PathFilter.create(path));
                    if (!treeWalk.next()) {
                        throw new IllegalStateException("Did not find expected file");
                    }

If the TreeWalk reaches to an end and does not find the specified skill path then it returns anIllegal State Exception with an message saying did not found the file on that commit ID.

       ObjectId objectId = treeWalk.getObjectId(0);
       ObjectLoader loader = repository.open(objectId);
       OutputStream output = new OutputStream();
       loader.copyTo(output);

And then one can the loader to read the file. From the treeWalk we get the object and create an output stream to copy the file content in it. After that we create the JSON and put the OutputStream object as as String in it.

       json.put("file",output);

This Servlet can be seen working api.susi.ai: http://api.susi.ai/cms/getFileAtCommitID.json?model=general&group=knowledge&language=en&skill=bitcoin&commitID=214791f55c19f24d7744364495541b685539a4ee

Resources

JGit Documentation: https://eclipse.org/jgit/documentation/

JGit User Guide: http://wiki.eclipse.org/JGit/User_Guide

JGit Repository access: http://www.codeaffine.com/2014/09/22/access-git-repository-with-jgit/

JGit Github: https://github.com/eclipse/jgit

Continue ReadingGetting SUSI Skill at a Commit ID

Stripe Authorization In Open Event API Server

The Open Event System supports payments through stripe. Stripe is a suite of APIs that powers commerce for businesses of all sizes. This blogpost covers testing of Stripe Authorization Schema and endpoints in the API Server.

The Stripe Authorization class provides the following endpoints:

'/stripe-authorization'
'/stripe-authorization/<int:id>'
'/events/<int:event_id>/stripe-authorization'
'/events/<event_identifier>/stripe-authorization'


In the pull request made for adding documentation and tests, these two endpoints were removed:

'stripe_authorization_list',
'/events/<int:event_id>/stripe-authorization',
'/events/<event_identifier>/stripe-authorization'

This is because each event can have only one stripe authorization, so there can not exist a list of stripe authorization objects related to an event.

The ‘stripe_authorization_list’ endpoint is made POST only. This is because Open Event does not allow individual resources’ list to be accessible. Since, there is no endpoint which returns a list of Stripe Authorizations the StripeAuthorizationList(ResourceListis removed.

The ResourceDetail class was modified to add a query to support  results from ‘/events/<int:event_id>/stripe-authorization’ endpoint suThe view_kwargs for the detail endpoint has to contain the resource id, so event_id from view_kwags is used to get the id for stripe authorization.

stripe_authorization = self.session.query(StripeAuthorization).filter_by(event_id=view_kwargs['event_id']).one()
view_kwargs['id'] = stripe_authorization.id

Writing Test for Documentation

(Tests for the /events/1/stripe-authorization  is described here, for others please refer to links in additional references.)

To test the  /events/1/stripe-authorization endpoint for GET, we first insert a Stripe Authorization object into the database which will then be retrieved by the GET request and then compared with the expected response body from the documentation file.

Since stripe-auth has a required relationship with event class, an event must also exist for strie auth object to be created. The event is also required because the endpoint ‘events/’ expects an event object to exist. The StripeAuthorizationFactory takes care of this with event as a RelatedFactory. So when a StripeAuthorization object is inserted, an event is created first and passed as the required relationship to stripe_list_post endpoint.

The event is related to the stripe object by setting event_id = 1 in the factory.

Adding the pre-test hook for GET:

@hooks.before("StripeAuthorization > Stripe Authorization for an Event > Get Stripe Authorization Details of an Event")
def event_stripe_authorization_get_detail(transaction):
   """
   GET /events/1/stripe-authorization
   :param transaction:
   :return:
   """
   with stash['app'].app_context():
       stripe = StripeAuthorizationFactory()
       db.session.add(stripe)
       db.session.commit()


The expected response for this request can be found
 here.

Additional References:

Continue ReadingStripe Authorization In Open Event API Server

Handling Errors While Parsing the yaml File in Yaydoc

Yaydoc, our automatic documentation generator uses a yaml file to read a user’s configuration. The internal configuration parser basically converts the yaml file to a python dictionary. Then, it serializes the values of that dictionary using a custom serialization format. From there it associates those values with environment variables which are then passed to bash scripts for various tasks such as deployment, generation, etc.. Some of those environment variables are again passed to another python layer which interacts with sphinx where they are deserialized before use. This whole system works pretty well for our use cases.

Now let’s assume a user adds a yaml file where they have a malformed section in the file. For example, to specify a theme, one needs to add the following to the yaml file.

build:
  theme:
    name: sphinx_fossasia_theme

But our user has the following in their yaml file.

build:
  theme: sphinx_fossasia_theme

Now this will raise an error as we expect a dictionary as a value for the key ‘theme’ but we got a string. Now how do we handle such cases without ignoring the entire file as that would be too much of a penalty for such a small mistake? One approach would have been to wrap each call to connect with a bunch of try-catch but that would render the code unreadable as the initial motivation for implementing the connect method was to abstract the internal implementation so that other contributors who may not be well versed with python can also easily add config options without needing to learn a bunch of python constructs.

So, what we did was that, while merging the dictionary containing default options and the dictionary containing the user preferences, we check whether the default has the same data type as that of the incoming value. If they are, It’s deemed safe to merge. There are certain relaxations though, like if the current type is a list, then the incoming value can be of any time as that can always be converted to a list of a single element. This is required to support the following syntax.

key:
  - value
key: value

The above two blocks are equivalent due to the above-mentioned approach although the type is different.

Now, after this pre-validation step is over we can ensure that the if the assumed type for a key is let’s say a dictionary, then it would be a dictionary. Hence no type errors would be raised like trying to access a dict method for another object, say a string which happened with the earlier implementation. After this, an extra parameter was added to the connect method to which we can now pass a validation function which if returns false, those values would be ignored. Usage of this feature has been implemented to a small level where we validate the links to subprojects and if they look like a valid github repo only then will they be included. Note that their existence is not checked. Only a regex based validation is performed.

It was also important to notify the user about these events when we detect that a specific section is invalid and provide informative and helpful error messages without failing the build. Hence proper error messages were also added which were informative so that the user knows exactly which section is to blame. This is similar to compilers where the error message is crucial to debug a certain piece of code.

Resources

Continue ReadingHandling Errors While Parsing the yaml File in Yaydoc