SUSI.AI – Page 17 – blog.fossasia.org

Getting SUSI Skill at a Commit ID

Post author:dynamitechetan
Post published:August 27, 2017
Post category:FOSSASIA
Post comments:0 Comments

Susi Skill CMS is a web app to edit and create new skills. We use Git for storing different versions of Susi Skills. So what if we want to roll back to a previous version of the skill? To implement this feature in Susi Skill CMS, we needed an API endpoint which accepts the name of the skill and the commit ID and returns the file at that commit ID.

In this blog post I will tell about making an API endpoint which works similar to git show.

First we will accept all the request parameters from the GET request.

        String model_name = call.get("model", "general");
        String group_name = call.get("group", "Knowledge");
        String language_name = call.get("language", "en");
        String skill_name = call.get("skill", "wikipedia");
        String commitID  = call.get("commitID", null);

In this we get the model name, category, language name, skill name and the commit ID. The above 4 parameters are used to make a file path that is used to find the location of the skill in the Susi Skill Data repository.

This servlet need CommitID to work and if commit ID is not given in the request parameters then we send an error message saying that the commit id is null and stop the servlet execution.

    Repository repository = DAO.getRepository();
    ObjectId CommitIdObject = repository.resolve(commitID);

Then we get the git repository of the skill from the DAO and initialize the repository object.

From the commitID that we got in the request parameters we create a CommitIdObject.

   (RevWalk revWalk = new RevWalk(repository)) {
   RevCommit commit = revWalk.parseCommit(CommitIdObject);
   RevTree tree = commit.getTree();

Now using commit’s tree, we will find the find the path and get the tree of the commit.

From the TreeWalk in the repository we will set a filter to find a file. This searches recursively for the files inside all the folders.

                revWalk = new RevWalk(repository)) {
                try (TreeWalk treeWalk = new TreeWalk(repository)) {
                    treeWalk.addTree(tree);
                    treeWalk.setRecursive(true);
                    treeWalk.setFilter(PathFilter.create(path));
                    if (!treeWalk.next()) {
                        throw new IllegalStateException("Did not find expected file");
                    }

If the TreeWalk reaches to an end and does not find the specified skill path then it returns anIllegal State Exception with an message saying did not found the file on that commit ID.

       ObjectId objectId = treeWalk.getObjectId(0);
       ObjectLoader loader = repository.open(objectId);
       OutputStream output = new OutputStream();
       loader.copyTo(output);

And then one can the loader to read the file. From the treeWalk we get the object and create an output stream to copy the file content in it. After that we create the JSON and put the OutputStream object as as String in it.

       json.put("file",output);

This Servlet can be seen working api.susi.ai: http://api.susi.ai/cms/getFileAtCommitID.json?model=general&group=knowledge&language=en&skill=bitcoin&commitID=214791f55c19f24d7744364495541b685539a4ee

Resources

JGit Documentation: https://eclipse.org/jgit/documentation/

JGit User Guide: http://wiki.eclipse.org/JGit/User_Guide

JGit Repository access: http://www.codeaffine.com/2014/09/22/access-git-repository-with-jgit/

JGit Github: https://github.com/eclipse/jgit

API to List All Users on SUSI.AI

Post author:DravitLochan
Post published:August 27, 2017
Post category:FOSSASIA GSoC SUSI.AI
Post comments:0 Comments

In this blog, I discuss how the SUSI server helps in listing out all the users registered on it. The only role Susi server plays is, Whenever it receives a request at

http://api.susi.ai/aaa/getUsers.json

The server evaluate the parameters in the request, validates them and notify the user accordingly. API needs 2 parameters, out of which access-token is a necessary. 2nd parameter has to be one from the given list :

Parameter Data type

getPageCount boolean
GetUserCount boolean
Page integer

On the basis of this 2nd parameter, server gets to know what does the client with given access-token is requesting. Server evaluates the access-token and validates that if the access token belongs to a user with user role atleast ADMIN, then the request is valid and proceed further with fetching the data in next step. Otherwise, server responds with error code “401” and error message “Base user role not sufficient”. It is advisable for clients that before redirecting users to admin panel or any other service, Please hit

http://api.susi.ai/aaa/showAdminService.json

And check that whether the user logged in is allowed to access the admin panel or not. The servlet /showAdminService.json is quite easy to understand for even those new to programming.

Coming back to our topic, by now, server knows that this client is authorized to access the user list. But what all information does server needs to provide? In response to this request, server encodes following attributes in the JSON Array {which is part of JSON object} and sends it to user :

Attribute Description

Name Email-Id of the user
Anonymous Is this user anonymous or not
User Role User Role of the user
Confirmed User has verified account or not
Last Login IP Last IP from which login was requested
Last Login Time Time when last login request was made
Signup Time When did the user signed up

First things first, check if enough parameters are provided or not. If not, respond with error stating “Bad Request. No parameter present”. Otherwise, server does a general iteration which has to be done irrespective of the 2nd parameter.

First of all, get a list of all the authorized users using getAuthorizedClients method of Data Access Object class. This method picks up all the keys from authorized file {which are nothing but identification of clients from which requests are received}. Though it, skips those key which are host addresses (which can not be used to identify a user), it does includes all the email ids {which are obvious identification of users}.

public static Collection<ClientIdentity> getAuthorizedClients() {
		ArrayList<ClientIdentity> i = new ArrayList<>();
		for (String id: authorization.keys()) {
		    if(id.contains("host"))
		        continue;
			i.add(new ClientIdentity(id));
		}
		return i;
	}

In next steps, the collection is converted to suitable data types over which iterations are easy and can be converted to JSON objects and Arrays easily. After this, server evaluates which parameter is requested in the request. Let us pick each case one by one for simplicity.

Client has requested number of pages in the request.

Server finds the size of keysArray {one of the object containing list of all the users}. Basic Mathematics to find out how many pages would be formed if size of each page is 50 elements and total elements are given.

if (call.get("getPageCount", false) == true) {
            int pageCount = keysArray.length % 50 == 0 ? (keysArray.length / 50) : (keysArray.length / 50) + 1;
            result.put("pageCount", pageCount);
            result.put("accepted", true);
            result.put("message", "Success: Fetched count of pages");
            return new ServiceResponse(result);
        }

User count is requested

Simply return sizeof list which has list of all the users. List to be used can be anyone from authorized, keysArray or any other derivative of authorized collection. Code is quite easy.

List of users on any page is requested.

Get the page number and after applying unitary maths, you will figure out the elementary of this.

for (Client client : authorized) {
                JSONObject json = client.toJSON();
                ClientIdentity identity = new ClientIdentity(ClientIdentity.Type.email, client.getName());
                Authorization authorization = DAO.getAuthorization(identity);
                UserRole userRole = authorization.getUserRole();
                json.put("userRole", userRole.toString().toLowerCase());
                userList.add(json);
            }

If any other attribute that is required, it’s encoding will be done here. For example, to get user role of a user, generate a client identity followed by retrieval of user role from it. Encode it and send back to user.

Other details like last login IP, last login time and signup time are also fetched from respective files.

Resources

Post by Tutorials Point on How to get size of a collection.
Post by Wikipedia : What is DAO?

Change Password for SUSI Accounts Using Access Token and Email-ID

Post author:DravitLochan
Post published:August 27, 2017
Post category:API FOSSASIA GSoC SUSI.AI
Post comments:0 Comments

In this blog, I discuss how the SUSI server synchronizes with SUSI Accounts and SUSI webchat for users to Change Password. When a user logs in, the clients store the email id of the user along with the access token in cookies. These are stored once the client gets a positive login response from the server. Both of these are required at the time of making the final call. Web clients store the email id and access token in the following way.

cookies.set('loggedIn', loggedIn, { path: '/', maxAge: time });
cookies.set('emailId', email, { path: '/', maxAge: time });

First, the client has to ask the user to enter their current password. A javascript test is used to validate that at least 6 characters must be entered by the user. A similar test is run on the new password. But while confirming the password, client checks whether the user has entered the same password as new password or not. These are just the basics. In next stage (which is achieved only when all the above conditions are met), client encodes the email id (which it gets from cookies), current password, new password and the access token (which it again extracts from cookies).

Now, Client just has to make an ajax request to the server. The response is processed and sent back to the client. Let us now look at PasswordChange Servlet.

The base user role is defined as USER. Initial steps of the servlet are to extract the values form the request it receives. The values extracted from the request are in turn used to make a client’s identity. Before that, server checks if current and new password have same values or not. If not, then server returns a JSON response to user stating, “Your current password and new password matches”. Otherwise, it will continue its control flow as it is. Look at the code snippet below:

if(password.equals(newpassword)){
            result.put("message", "Your current password and new password matches");
            result.put("accepted", false);
            return new ServiceResponse(result);
        }

The reader here may think that they have discovered a hack. But they have not. Why? Because this is just the first step. In later stages, the hash of passwords are used to match to see whether the passwords match or not. To obtain a proper client identity, first a Client credentials object is made with support from the email id which is received in ‘changepassword’ attribute. Using the ClientCredentials object made above, an object of Authentication class is made. This object uses a method defined in its class to return a valid client identity. Using the client identity, value of password hash is extracted from the database along with the salt used to hash the password. If any error is encountered while extracting the client’s password hash value and/or salt value, an error is thrown towards the client, with a message stating “invalid credentials”.

ClientCredential pwcredential = new ClientCredential(ClientCredential.Type.passwd_login, useremail);
            Authentication authentication = DAO.getAuthentication(pwcredential);
            ClientCredential emailcred = new ClientCredential(ClientCredential.Type.passwd_login,
                authentication.getIdentity().getName());
            ClientIdentity identity = authentication.getIdentity();
            String passwordHash;
            String salt;

            try {
                passwordHash = authentication.getString("passwordHash");
                salt = authentication.getString("salt");
            } catch (Throwable e) {
                Log.getLog().info("Invalid password try for user: " + identity.getName() + " from host: " + post.getClientHost() + " : password or salt missing in database");
                result.put("message", "invalid credentials");
                throw new APIException(422, "Invalid credentials");
            }

Using the same salt value that was used earlier, a hash for password entered by the user will be generated which now matches the previous value. This is the point where the hack you were thinking you found, failed. Again the server throws an error message if user’s credential did not match. Passwords are hard to handle and easy to guess. So here we have used quite many tests before changing them. Users are not allowed to use their email id as a password as well.

If the server is clear on all the above facts and tests, It finally generates a new hashed value of the password received in the parameter ‘newpassword’ and replaces the old hash value with the new one. To notify the clients that password change exited with a success response, it sends a JSON object with message “Your password has been changed!” and accepted flag set to true.

if (DAO.hasAuthentication(emailcred)) {
                    Authentication emailauth = DAO.getAuthentication(emailcred);
                    String newsalt = createRandomString(20);
                    emailauth.remove("passwordHash");
                    emailauth.put("passwordHash", getHash(newpassword, salt));
                    Log.getLog().info("password change for user: " + identity.getName() + " via newpassword from host: " + post.getClientHost());
                    result.put("message", "Your password has been changed!");
                    result.put("accepted", true);
                }

Additional Resources:

Stackoverflow, post by user erickson about How can I hash a password in java
Blog post on blog.sodhanalibrary.com by Srinivas Dasari about How to implement Change Password in a JAVA server

Wikipedia article: What is DAO?

Implementation of SUSI.AI Signup Service

Post author:DravitLochan
Post published:August 27, 2017
Post category:API FOSSASIA GSoC SUSI.AI
Post comments:0 Comments

In this blog, I discuss the complete implementation of SUSI server when a user signs up with SUSI. Since public signup is enabled, minimal user role to access the servlet is ANONYMOUS. Given below is the API endpoint at which a signup request has to be made.

http://api.susi.ai/aaa/signup.json

Here we do not have much options unlike Login service. . Requests can be made in following ways with different parameters:

http://api.susi.ai/aaa/signup.json?signup=test@fossasia.com&password=Rest@123

http://api.susi.ai/aaa/signup.json?getParameters=true

http://api.susi.ai/aaa/signup.json?access_token={30 characters long access token}&validateEmail=test@fossasia.com&request_session=true

First one is the request which client makes, asking the server to register the user with email id test@fossasia.com and password as Reset@123. On receiving a request which has parameters signup and password, server looks back in the database and verifies that a user with same email id exists or not. If email id already exists, server throws an error stating “email already taken”. Given below is a small code snippet for better understanding.

ClientCredential credential = new ClientCredential(ClientCredential.Type.passwd_login, signup);
		Authentication authentication = DAO.getAuthentication(credential);

		if (authentication.getIdentity() != null) {
			throw new APIException(422, "email already taken");
		}

If client is not found in the database, a new identity is generated for the user and is set in authentication.json file. Next, a random salt string is generated which acts as salt to hash the user’s password. Since the same salt will be required at the time of matching the password for validation, we store both the salt and password hash in authentication.json file.

Now, if the database is going to encounter it’s first user, Automatically, BUREAUCRAT (Role with maximum power) will be assigned as his/her user role. To do this, we first get a list of all the users with the help of DAO class and count number of users present in it. Take a look at the code given below,

Authorization authorization = DAO.getAuthorization(identity);
		Collection<ClientIdentity> authorized = DAO.getAuthorizedClients();
		List<String> keysList = new ArrayList<String>();
		authorized.forEach(client -> keysList.add(client.toString()));
		String[] keysArray = keysList.toArray(new String[keysList.size()]);
		if(keysArray.length == 1) {
			authorization.setUserRole(UserRole.BUREAUCRAT);
		} else {
			authorization.setUserRole(UserRole.USER);
		}

In further steps, a random token is generated and marked as the verification token for the user who just signed-up. Other attributes for the token are also set like expiry time. Finally, account verification mail’s template is picked from “conf/templates/verification-mail” file and used as the body for the user verification email.

The second type of request is quite simple which is just to request a regular expression from the server to give proper information to the user. It’s code is quite easy and self understandable.

Third type of request is sent to the server when user clicks on the link received in the account verification email. Taking a closer look at the account verification link, one could easily decode that the request would be redirected to /aaa/signup.json with parameters

Access_token : 30 characters long access token
validateEmail : user’s email id
request_session : true

Access_token makes up the unique id for the user. ValidateEmail is to make it easier for server to generate a client identity. Let us now understand how server picks up each parameter and activates the user’s account.

if (post.get("validateEmail", null) != null) {
			if((auth.getIdentity().getName().equals(post.get("validateEmail", null)) && auth.getIdentity().isEmail()) // the user is logged in via an access token from the email {

				ClientCredential credential = new ClientCredential(ClientCredential.Type.passwd_login,
						auth.getIdentity().getName());
Authentication authentication =     DAO.getAuthentication(credential);

				if (authentication.getIdentity() == null) {
					authentication.delete();
					throw new APIException(400, "Bad request");
				}
				authentication.put("activated", true);
				result.put("accepted", true);
result.put("message", "You successfully verified your                                        account!");
				return new ServiceResponse(result);
			}
			throw new APIException(400, "Bad request"); // do not leak if user exists or not
		}

In the above code snippet, an object of ClientCredential class is generated and used to get authentication object for the user. If user is not found (name in authentication object is null), server throws error with error message “Bad Request”. Finally, If no error was encountered, server activates the user.

In getDefaultPermissions() function, permission for each user role is specified. All the users registered are allowed to register other users, but no user with anonymous user role can register anyone else except him/her.

Resources

Post by Tutorials Point on How to send email in JAVA
Post by Wikipedia on What is DAO?
Answer by Tomasz Nurkiewicz on StackoverFlow Need for server side account validation instead of client side

SUSI Account Verification – From JSON to Accounts

Post author:DravitLochan
Post published:August 27, 2017
Post category:FOSSASIA GSoC SUSI.AI
Post comments:0 Comments

In this blog post, I’ll be telling on how SUSI-server synchronizes with SUSI accounts front-end and enable users to have a SUSI verified account. Apart from how server processes the query and responds to it, I also discuss the role of SUSI accounts as a client in it.

Beginning with server, at the time of signup, a mail is sent to user’s email id {only is user is not already registered}. This email contains a link, clicking on which a request is sent to server to check if the combination of email id and the access_token passed matches or not. Let us quickly look at the email and decode it.

http://accounts.susi.ai/verify-account?access_token=hU86imBzXP3hcLgBgN6x4ShLuDopyH&validateEmail=dlochangupta@gmail.com&request_session=true

As you can see, the base URL is http://accounts.susi.ai and the route to which the user is redirected ‘/verify-account’. What does the rest of the parameters do?

access _token —> access token
validateEmail —> email id of the user
request_session —> boolean value to request a session

access_token contains a key which will be valid for 7 days (i.e. 604800 seconds) since the account was registered.

Since signing up and verifying user’s account, both share Signup.java servlet, As a first step, server checks if the request contains ‘validateEmail’ in the parameter list. If yes, AND it is not a null parameter, server is sure that it is a account verification request. Next it proceeds with checking if user has even registered or not. If not, user is notified with error code 400 and error message “Bad request”. Otherwise, by now server has found the client ID already and only steps left are to make “activated” attribute set to true. Once done successfully, server responds with a similar type of JSON response :

{
  "accepted": true,
  "message": "You successfully verified your account!",
  "session": {"identity": {
    "type": "email",
    "name": "test@fossasia.com",
    "anonymous": false
  }}
}

But you did not come across a similar type of JSON response. Did you? Here comes the role of client. If you replace http://accounts.susi.ai/verify-account with http://api.susi.ai/aaa/signup.json in the link you received, you will indeed find this JSON response. Since a user is redirected to http://accounts.susi.ai/verify-account with above mentioned 3 parameters, in componentDidMount() function, list of these parameters is extracted from component props.

componentDidMount() {
    const {
      accessToken, validateEmail, requestSession
    } = this.props;

This list was declared and initialized with some values initially so that if these are not found in the parameter list, client would be able to figure it out.

const urlPropsQueryConfig = {
  accessToken: { type: UrlQueryParamTypes.string, queryParam: 'access_token' },
  requessession: { type: UrlQueryParamTypes.string},
  validateEmail: { type: UrlQueryParamTypes.string },
};

static defaultProps = {
    token: 'null',
    validateEmail: 'null',
    requestSession: false,
  }

Additional Resources

Site: npmjs.com on How to extract url query parameters from a link? (official documentation)
Site: daveceddia.com on How to make an AJAX call in ReactJS. Blog post by Dave Ceddia

Implementing Skill Listing in SUSI Android App using Nested RecyclerViews

Post author:chiragw15
Post published:August 27, 2017
Post category:Android API FOSSASIA GSoC SUSI.AI Tutorial
Post comments:0 Comments

SUSI Skills are rules that are defined in SUSI Skill Data repo which are basically the responses SUSI gives to the user queries. When a user queries something from the SUSI Android app, a query to SUSI Server is made which further fetches response from SUSI Skill Data and gives the response to the app. Similarly, when we need to list all skills, an API call is made to server to list all skills. The server then checks the SUSI Skill Data repo for the skills and then return all the required information to the app. Then the app displays all the information about the skill to user. User then can view details of each skill and then interact on the chat interface to use that skill. This process is similar to what SUSI Skill CMS does. The CMS is a skill wiki like interface to view all skills and then edit them. Though the app can not be currently used to edit the skills but it can be used to view them and try them on the chat interface.

API Information

For listing SUSI Skill groups, we have to call on /cms/getGroups.json

This will give you all groups in SUSI model in which skills are present. Current response:

{
  "session": {"identity": {
    "type": "host",
    "name": "14.139.194.24",
    "anonymous": true
  }},
  "accepted": true,
  "groups": [
    "Small Talk",
    "Entertainment",
    "Problem Solving",
    "Knowledge",
    "Assistants",
    "Shopping"
  ],
  "message": "Success: Fetched group list"
}

So, the groups object gives all the groups in which SUSI Skills are located.

Next comes, fetching of skills. For that the endpoint is /cms/getGroups.json?group=GROUP_NAME

Since we want all skills to be fetched, we call this api for every group. So, for example we will be calling http://api.susi.ai/cms/getSkillList.json?group=Entertainment for getting all skills in group “Entertainment”. Similarly for other groups as well.

Sample response of skill:

{
  "accepted": true,
  "model": "general",
  "group": "Shopping",
  "language": "en",
  "skills": {"amazon_shopping": {
    "image": "images/amazon_shopping.png",
    "author_url": "https://github.com/meriki",
    "examples": ["Buy a dress"],
    "developer_privacy_policy": null,
    "author": "Y S Ramya",
    "skill_name": "Shop At Amazon",
    "dynamic_content": true,
    "terms_of_use": null,
    "descriptions": "Searches items on Amazon.com for shopping",
    "skill_rating": null
  }},
  "message": "Success: Fetched skill list",
  "session": {"identity": {
    "type": "host",
    "name": "14.139.194.24",
    "anonymous": true
  }}
}

It gives all details about skills:

image
author_url
examples
developer_privacy_policy
author
skill_name
dynamic_content
terms_of_use
descriptions
skill_rating

Implementation in SUSI Android App

Skill Listing UI of Google Assistant

Skill Listing UI of SUSI SKill CMS

Skill Listing UI of SUSI Android App

The UI of skill listing in SUSI Android App is the mixture of UI of Skill listing in Google Assistant ap and SUSI Skill CMS. It displays skills in a beautiful manner with horizontal recyclerview nested in vertical recyclerview.

So, for implementing horizontal recyclerview inside vertical recyclerview, you need two viewholders and two adapters (one each for a recyclerview).

Let’s see the implementation.

1. First task is to fetch the information of groups in which skills are located. This line calls method in SkillListModel which then makes an API call to fetch groups.

skillListingModel.fetchGroups(this)

2. When the API call is succeeded, the below mentioned method is called which then calls a skillListingModel.fetchSkills(groups[0], this) which fetches the skills located in group[0] group.

override fun onGroupFetchSuccess(response: Response<ListGroupsResponse>) {
   if (response.isSuccessful && response.body() != null) {
       groupsCount = response.body().groups.size
       groups = response.body().groups
       skillListingModel.fetchSkills(groups[0], this)
   } else {
       skillListingView?.visibilityProgressBar(false)
       skillListingView?.displayErrorDialog()
   }
}

3. When API call for fetching skills in group[0] succeeds, the count value is increased and then skills in group[1] are fetched and so on.

override fun onSkillFetchSuccess(response: Response<ListSkillsResponse>, group: String) {
   if (response.isSuccessful && response.body() != null) {
       skills.add(Pair(group, response.body().skillMap))
       count++
       if(count == groupsCount) {
           skillListingView?.visibilityProgressBar(false)
           skillListingView?.updateAdapter(skills)
       } else {
           skillListingModel.fetchSkills(groups[count], this)
       }
   } else {
       skillListingView?.visibilityProgressBar(false)
       skillListingView?.displayErrorDialog()
   }
}

4. When skills in all groups are fetched, the data in adapter is updated using skillGroupAdapter.notifyDataSetChanged()

override fun updateAdapter(skills: ArrayList<Pair<String, Map<String, SkillData>>>) {
   this.skills.clear()
   this.skills.addAll(skills)
   skillGroupAdapter.notifyDataSetChanged()
}

5. The data is set to the layout in two adapters made earlier. The following is the code to set the group name and adapter to horizontal recyclerview. This is the GroupAdapter to set data to row item in vertical recyclerview.

override fun onBindViewHolder(holder: GroupViewHolder?, position: Int) {
   if(skills[position].first != null)
       holder?.groupName?.text = skills[position].first
   holder?.skillList?.setHasFixedSize(true)
   val mLayoutManager = LinearLayoutManager(context, LinearLayoutManager.HORIZONTAL, false)
   holder?.skillList?.layoutManager = mLayoutManager
   holder?.skillList?.adapter = SkillListAdapter(context, skills[position])
}

6. Similarly, the data of each individual element in the horizontal recyclerview is set in the skillAdapter. The data set are title, examples, description and image. We have used Picasso library to load images from the URL.

override fun onBindViewHolder(holder: SkillViewHolder?, position: Int) {
   val skillData = skillDetails.second.values.toTypedArray()[position]

   if(skillData.skillName == null || skillData.skillName.isEmpty()){
       holder?.skillPreviewTitle?.text = context.getString(R.string.no_skill_name)
   } else {
       holder?.skillPreviewTitle?.text = skillData.skillName
   }

   if( skillData.descriptions == null || skillData.descriptions.isEmpty()){
       holder?.skillPreviewDescription?.text = context.getString(R.string.no_skill_description)
   } else {
       holder?.skillPreviewDescription?.text = skillData.descriptions
   }

   if(skillData.examples == null || skillData.examples.isEmpty())
       holder?.skillPreviewExample?.text = StringBuilder("\"").append("\"")
   else
       holder?.skillPreviewExample?.text = StringBuilder("\"").append(skillData.examples[0]).append("\"")

   if(skillData.image == null || skillData.image.isEmpty()){
       holder?.previewImageView?.setImageResource(R.drawable.ic_susi)
   } else {
       Picasso.with(context.applicationContext).load(StringBuilder(imageLink)
               .append(skillDetails.first.replace(" ","%20")).append("/en/").append(skillData.image).toString())
               .fit().centerCrop()
               .into(holder?.previewImageView)
   }
}

Summary

So, this blog talked about how the Skill Listing feature in SUSI Android App is implemented. This included how a network call is made, logic for making different network calls, making a nested horizontal recyclerview inside vertical recyclerview, etc. So, If you are looking forward to contribute to SUSI Android App, this can help you a little. But if not so, this may also help you in understanding and how you can implement nested recyclerviews similar to Google Play Store.

References

To know about servlets https://en.wikipedia.org/wiki/Java_servlet
To see how to implement one https://www.javatpoint.com/servlet-tutorial
To see how to make network calls in android using Retrofit https://guides.codepath.com/android/Consuming-APIs-with-Retrofit
To see how to implement Horizontal recyclerView inside Vertical recyclerView http://android-pratap.blogspot.in/2015/12/horizontal-recyclerview-in-vertical.html
To see how to implement custom RecyclerView Adapter https://www.survivingwithandroid.com/2016/09/android-recyclerview-tutorial.html

How to Parse HTML Tags and Anchor Clickable Links in SUSI Android App

Post author:amitiwary999
Post published:August 22, 2017
Post category:FOSSASIA GSoC
Post comments:0 Comments

Html tags are used to define how contents of a webpage should be formatted and displayed. Sometimes the SUSI answer type response contains some html tags but showing these html tags without parsing would distort the normal text flow in SUSI Android.

For the query ‘Ask me something’ SUSI’s reply is

“data”: [
     {
      “question”: “Which soccer team won the Copa América 2015 Championship ? “,
     }]

In SUSI Android this message looks like

As you can see that showing html tags without parsing distort the normal text flow. So we need to parse html tags properly. We use Html class for this purpose. Html class is present in android.text package and you can import it in the class where you want to use it.

import android.text.Html

fromHtml method of Html class is used to parse html tags. But for API level less than 24 and equal to or greater than 24 we use different parameters in fromHtml method.

For API level less than 24 we used

Html.fromHtml(model.getContent())

But for API level equal to or greater than 24 we have to use

Html.fromHtml(model.getContent(), Html.FROM_HTML_MODE_COMPACT)

Here the second parameter is legacy flags which decides how text inside a tag will be shown after parsing.

In case of Html.fromHtml(String string) legacy flag is by default FROM_HTML_MODE_LEGACY. It indicates that separate block-level elements with blank lines.

So after parsing html tags using fromHtml

But return type of fromHtml method is Spanned so if you need String then you have to convert it into string using toString() method.

Anchor action type in susi response contains link and text.

“link”: “https://www.openstreetmap.org/#map=13/1.2896698812440377/103.85006683126556”,
“text”: “Link to Openstreetmap: Singapore”

Here the text is the text we show in textview and link is used to show web content in the browser when user click on text. So first link and text are attached together like

“<a href=\”” +susiResponse.getAnswers().get(0).getActions().get(i).getAnchorLink() + “\”>”
+ susiResponse.getAnswers().get(0).getActions().get(1).getAnchorText() + “</a>”

Here text between the tag is description of link and after parsing we show this text in textview. It can be parsed using fromHtml method of Html class and textview is made clickable by

chatTextView.setMovementMethod(LinkMovementMethod.getInstance());

Resources

Main site link of Html class: https://developer.android.com/reference/android/text/Html.html
Blog link on how to show an Html string in an Android textview: http://alvinalexander.com/android/how-show-html-string-in-android-textview-webview

Reset Password Option in SUSI Android App

Post author:amitiwary999
Post published:August 22, 2017
Post category:FOSSASIA GSoC
Post comments:0 Comments

Login and signup are an important feature for some android apps like chat apps because the user will want to save and secure personal messages from others. In SUSI Android app we provide a token to a logged-in user for a limit period of time so that once the user logs in and someone else gets access to the device, then he/she can’t use the user account for a long period of time. It is a security provided from our side but the user also has to maintain some security. Cyber security risks have increased and hacking technologies have improved a lot in the past 10 years. So, using the same password for a long period of time absolutely puts your account security at risk. So to keep your account secure you should change/reset your password regularly. In this blog post, I will show you how reset password option is implemented in SUSI Android app.

Layout design for reset password

Reset password option is added in the setting. When the user clicks on reset password option a dialog box pops up. There are three textinputlayout boxes – each for the current password, new password and confirm password. I have used textinputlayout instead of simple edittext box because it helps user to show first “hint” and when user taps on, hint will come up with text over it as floating label so that the user can understand what to add in that box and also in case of error we can show that error to user.

Reset Password implementation

On clicking reset password option a dialog box appears in which user inserts the current password, new password and confirm password to confirm the new password. Before sending new password to the server we perform two checks

New password should not be empty and length of new password should be at least six.
New password and confirm password must be same.

if (!CredentialHelper.isPasswordValid(newPassword)) {

settingView?.passwordInvalid(Constant.NEW_PASSWORD)

return

}

if (newPassword != conPassword) {

settingView?.invalidCredentials(false, Constant.NEW_PASSWORD)

return

}

And when these two checks are passed we send “new password” to server.

Endpoint use to reset password is

http://api.susi.ai/aaa/changepassword.json?changepassword=your mail id&password=current password&newpassword=newpassword

As you can see it needed three parameters

changepassword: Your email id
password : Your current password
newpassword: Your new password

When user logs in, we save user’s email id so that the user doesn’t have to provide it again and again when the user wants to change the password.

utilModel.saveEmail(email)

The user provides current password and new password through dialog box. We used resetPassword method to reset the password. We send these three parameters to the server using resetPassword method and if the password changed successfully then server sends a message.

override fun resetPassword(password: String, newPassword: String, listener: ISettingModel.onSettingFinishListener) {

val email = PrefManager.getString(Constant.SAVE_EMAIL, null)

resetPasswordResponseCall = ClientBuilder().susiApi

.resetPasswordResponse(email,password,newPassword)

resetPasswordResponseCall.enqueue(object : Callback<ResetPasswordResponse> {

} )

}

We used retrofit library for network call and resetPasswordResponse is a model class using which we are retrieving server response.

Reference

Tutorial on how to use textinputlayout: https://code.tutsplus.com/tutorials/creating-a-login-screen-using-textinputlayout–cms-24168
Tutorial on how to build custom dialog box: http://android.pcsalt.com/create-alertdialog-with-custom-layout-using-xml-layout/
Github link of SUSI Server: https://github.com/fossasia/susi_server
Tutorial on how to use retrofit library using kotlin: https://segunfamisa.com/posts/using-retrofit-on-android-with-kotlin

Updating Skill Data Repository through SUSI Server

Post author:saurabhjn76
Post published:August 21, 2017
Post category:FOSSASIA GSoC SUSI.AI
Post comments:0 Comments

SUSI Skill Data is the storage place for susi skills. All the skills are stored in these repository follows schema Models -> Groups -> Languages -> Skill -> Intents. Using this, one can access any skill based on four tuples parameters model, group, language, skill. The SUSI skill CMS is an editor to write and edit skill easily. It follows an API-centric approach where the SUSI server acts as API server and a web front-end act as the client for the API and provides the user interface. In this blog we will see how SUSI server updates the skill data repository.

For implementing versioning of SUSI Skills, server uses JGit. JGit is an EDL licensed, lightweight, pure Java library implementing the Git version control system. To include a Gradle dependency to add JGit to the SUSI Server project.

compile 'org.eclipse.jgit:org.eclipse.jgit:4.6.1.201703071140-r'

Next we initialize the SUSI Skill Data repository in DAO.

 susi_skill_repo = new File(data_dir.getParentFile().getParentFile(), "susi_skill_data/.git");
repository = builder.setGitDir((DAO.susi_skill_repo))
           .readEnvironment() // scan environment GIT_* variables
           .findGitDir() // scan up the file system tree
           .build();

The code above opens our local git repository and creates an object “git”. Then we perform further operations on “git” object. Now we add our changes to “git”. This is similar to when we run any command using git. Next, to pull the latest changes we define a pull method catching the IOException,

public static void pull(Git git) throws IOException {
       try {
           PullResult pullResult = git.pull().call();
           MergeResult mergeResult = pullResult.getMergeResult();

           if (mergeResult!=null && mergeResult.getConflicts()!=null) {
               pullStatus =false;
               // we have conflicts send email to admin
               try {
                   EmailHandler.sendEmail(getConfig("skill_repo.admin_email",""), "SUSI Skill Data Conflicts", getConflictsMailContent(mergeResult));
               } catch (Throwable e) {
                e.printStackTrace();
               }

           } else {
               PushCommand push = git.push();
               push.setCredentialsProvider(new UsernamePasswordCredentialsProvider(getConfig("github.username", ""), getConfig("github.password","")));
               push.call();
           }
           
       } catch (GitAPIException e) {
           throw new IOException (e.getMessage());
       }

We check if there are any merge conflicts while merging the latest changes and local commits, if there are any we send the email to administrator to resolve them manually. Otherwise we push the commits to update the local git repository. The credentials currently are stored in config.properties file, and is used using the getConfig method.
To pull Susi_SKill_Data every minute we add a thread in DAO initialization method.

 log("Starting Skill Data pull thread");
       Thread pullThread = new Thread() {
           @Override
           public void run() {
               while (DAO.pullStatus) {
                   try {
                       Thread.sleep(getConfig("skill_repo.pull_delay", 60000));
                   } catch (InterruptedException e) {
                       break;
                   }
                   try {
                       DAO.pull(DAO.getGit());
                   } catch (Exception e) {
                       DAO.pullStatus =false;
                       Log.getLog().warn("SKILL PULL THREAD", e);
                   }
               }
           }
       };
       pullThread.start();

The thread sleeps for a default of 60 seconds, and updates the skill_data until any exception is faced. In case of exception the threads stops and administrator is informed about the git status.

This is how server pulls skill data every minute. To add the metadata to the skill visit susi_skill_data, the storage place for susi skills. For more information and complete code take a look at Susi server and join gitter chat channel for discussions.

Resources

JGit documentation : https://eclipse.org/jgit/documentation/
The source code SUSI Server : https://github.com/fossasia/susi_server
Read about servlet interface – https://www.javatpoint.com/Servlet-interface
Read about DAO- https://en.wikipedia.org/wiki/Data_access_object

Implementing Skill Page in SUSI Skill CMS

Post author:saurabhjn76
Post published:August 21, 2017
Post category:FOSSASIA GSoC SUSI.AI
Post comments:0 Comments

The SUSI skill CMS is an editor to write and edit skill easily. It follows an API-centric approach where the SUSI server acts as API server and a web front-end act as the client for the API and provides the user interface. A skill is a set of intents. One text file represents one skill, it may contain several intents which all belong together. All the skills are stored in SUSI Skill Data repository and the schema is Models -> Groups -> Languages -> Skill -> Intents. Using this, one can access any skill based on four tuples parameters model, group, language, skill.

A skill format looks like following,

::name <Skill_name>
::author <author_name>
::author_url <author_url>
::description <description> 
::dynamic_content <Yes/No>
::developer_privacy_policy <link>
::image <image_url>
::terms_of_use <link>

#Intent
User query1 query2 quer3....
!example:<The question that should be shown in public skill displays>
!expect:<The answer expected for the above example>
Answer for the user query

Susi Skill class in Susi Server provides parser methods for sets of intents, given as text files. It reads the skill line by line. The SusiMind class process this JSON and stores the metadata in a map of skill path and data, which provides the API endpoint for getting the skill the metadata . http://api.susi.ai/cms/getSkillMetadata.json?model=general&group=Knowledge&language=en&skill=cars. It takes four parameters, model, group, language, and skill. The endpoint returns null value if the skill metadata field is not present in the skill. The following is the response for model=general, group=Knowledge, language =en and skill =cars.

"skill_metadata": {
    "model": "general",
    "group": "Knowledge",
    "language": "en",
    "developer_privacy_policy": null,
    "descriptions": "A skill to tell user about cars made by a car manufacturer",
    "image": "images/cars.png",
    "author": "Uday Theja",
    "author_url": "https://github.com/uday96",
    "skill_name": "Cars",
    "terms_of_use": null,
    "dynamic_content": true,
    "examples": [
      "Know any car models manufactured by ford",
      "Know any car models manufactured by ford in the year 2016",
      "Know any car models manufactured in 2016 by ford"
    ]
  },

Image is the path to image of the skill, stored in skill data repository and dynamic content represents if the skill’s response changes based on the user inputs.
To make the required API call we first need the value of 4 parameters. We will get these as props from browse skill route.

<Link key={el}
   to={{
        pathname: '/skillPage',
state: { url: url, element: el, name: el, modelValue: self.state.modelValue, groupValue:self.state.groupValue,        languageValue:self.state.languageValue}
                             }}>

Using these props we will append the base url with all the four parameters.

 let baseUrl = 'http://api.susi.ai/cms/getSkillMetadata.json';
 url = baseUrl + '?model=' + modelValue + '&group=' + groupValue + '&language=' + languageValue + '&skill=' + name;

Next we will make a ajax call and get the data updated.

 $.ajax({
                url: url,
                jsonpCallback: 'pc',
                dataType: 'jsonp',
                jsonp: 'callback',
                crossDomain: true,
                success: function (data) {
                    self.updateData(data.skill_metadata)
                }
            });

To display data on Skill CMS we will use Material UI components. Before rendering we will check the each state if it is null , we will display the default values for it.

{this.state.imgUrl === undefined?
                            <CircleImage name={this.state.skill_name.toUpperCase()} size="250"/>:
                            <img className="avatar-img" alt="Thumbnail" src={this.state.imgUrl}/>

If the image is not present in the skill, we display the placeholder showing the name of the skill.

Otherwise, the image stored in skill_data repository is shown.

Similarly, we will display the other components using material-ui. Next, we add an edit button for editing the skill.

To make edit button we use floating action button. We link the button to skillEditor route and pass the URL and skillname as props.

  <Link to={{
           pathname: '/skillEditor',
           state: { url: urlCode, name:name, }
   }}>

SkillEditor components uses getSkill.json endpoint and using these props make the ajax call to the endpoint like: http://api.susi.ai/cms/getSkill.json?model=general&group=Entertainment&language=en&skill=<skill_name>.

This is how skill Page is implemented in Susi skill CMS, To add more skills or modify the existing ones visit skills.susi.ai. For more information and complete code take a look at Susi server and Susi_Skill_cms and join gitter chat channel for discussions.

Resources

Read more about Servlets – https://www.tutorialspoint.com/servlets/
The source code –https://github.com/fossasia/susi_skill_cms/tree/master/src/components/SkillPage
Read about servlet interface – https://www.javatpoint.com/Servlet-interface
Read merits of Material UI- https://thenextweb.com/dd/2015/11/10/what-are-the-real-merits-of-material-design/#.tnw_hnWuEF06