Open Event API Server: Implementing FAQ Types

In the Open Event Server, there was a long standing request of the users to enable the event organisers to create a FAQ section.

The API of the FAQ section was implemented subsequently. The FAQ API allowed the user to specify the following request schema

 "data": {
   "type": "faq",
   "relationships": {
     "event": {
       "data": {
         "type": "event",
         "id": "1"
   "attributes": {
     "question": "Sample Question",
     "answer": "Sample Answer"


But, what if the user wanted to group certain questions under a specific category. There was no solution in the FAQ API for that. So a new API, FAQ-Types was created.

Why make a separate API for it?

Another question that arose while designing the FAQ-Types API was whether it was necessary to add a separate API for it or not. Consider that a type attribute was simply added to the FAQ API itself. It would mean the client would have to specify the type of the FAQ record every time a new record is being created for the same. This would mean trusting that the user will always enter the same spelling for questions falling under the same type. The user cannot be trusted on this front. Thus the separate API made sure that the types remain controlled and multiple entries for the same type are not there.

Helps in handling large number of records:

Another concern was what if there were a large number of FAQ records under the same FAQ-Type. Entering the type for each of those questions would be cumbersome for the user. The FAQ-Type would also overcome this problem

Following is the request schema for the FAQ-Types API

 "data": {
   "attributes": {
     "name": "abc"
   "type": "faq-type",
   "relationships": {
     "event": {
       "data": {
         "id": "1",
         "type": "event"



  • FAQ to FAQ-type is a many to one relation.
  • A single FAQ can only belong to one Type
  • The FAQ-type relationship will be optional, if the user wants different sections, he/she can add it ,if not, it’s the user’s choice.

Related links

List all the Users Registered on SUSI.AI

In this blog, I’ll be telling on how SUSI admins can access list of all the registered users from SUSI-server. Following this, they may modify/edit user role of any registered user.

What is User Role?

A UserRole defines the servlet access right. Not all users are allowed to access all the data and services. For  example, To list all the users, minimal user role expected is ADMIN. This classification of users are inspired by the wikipedia User Access Levels, see querying SUSI, Users are classified into 7 different categories, namely :

  • BOT
  • USER  

* Please see that these are as of the date of publish of this blog. These are subject to change, which is very unlikely.

All the users who are not logged in but interacting with SUSI are anonymous users. These are only subject to chat with SUSI, login, signup or may use forgot password service. Once a user login to the server, a token is generated and sent back to client to maintain the identity, hence acknowledging them. Privileged users are those who have special rights with them. These are more like moderators with much special rights than any other user. At the top level of the hierarchy are the admins. These users have more rights than anyone. They can change role of any other user, override decision of any privileged user as well.

Let us now look at the control flow of this.

First things first, make a component of User List in the project. Let us name it ListUsers and since it has to be accessible by those users who possess ADMIN rights, you will find it enclosed in Admin package in components folder. Open up

index.js file, import Listusers component  and add route to it in the following way :

...//other import statements
import ListUser from "./components/Admin/ListUser/ListUser";
...//class definition and other methods
<Route path="/listUser" component={ListUser}/>
//other routes defined

Find a suitable image for “List Users” option and add the option for List Users in static appbar component along with the image. We have used Material UI’s List image in our project.

...// other imports

import List from 'material-ui/svg-icons/action/list';

Class and method definition

<MenuItem primaryText="List Users"
          containerElement={<Link to="/listUser" />}

...//other options in top right corner menu

Above code snippet will add an option to redirect admins to ‘/listUsers’ route. Let us now have a closer look at functionality of both client and server. By now you must have known what ComponentDidMount does. {If not, I’ll tell you. This is a method which is given first execution after the page is rendered. For more information, visit this link}. As mentioned earlier as well that this list will be available only for admins and may be even extended for privileged users but not for anonymous or any other user, an AJAX call is made to server in ComponentDidMount of ‘listuser’ route which returns the base user role of current user. If user is an Admin, another method, fetchUsers() is called.

let url;
        url = "";
            url: url,
            dataType: 'jsonp',
            jsonpCallback: 'py',
            jsonp: 'callback',
            crossDomain: true,
            success: function (response) {
                if (response.userRole !== "admin") {
                    console.log("Not an admin")
                } else {

In fetchUsers method, an AJAX call is made to server which returns username in JSONArray. The response looks something likes this :

	"users" : {
		"email:""[email protected]",
"Username":["[email protected]", "[email protected]"...]

Now, only rendering this data in a systematic form is left. To give it a proper look, we have used material-ui’s table. Import Table, TableBody, TableHeader,

   TableHeaderColumn, TableRow, TableRowColumn from material-ui/table.

In fetchUsers method, response is catched in data Oblect. Now the keys are extracted from the JSON response and mapped with an array. Iterating through array received as username array, we get list of all the registered users. Now, popuulate the data in the table you generated.

return (
                        <TableRow key={i}>
                            <TableRowColumn> </TableRowColumn>
                            <TableRowColumn> </TableRowColumn>
                            <TableRowColumn> </TableRowColumn>
                            <TableRowColumn> </TableRowColumn>

Above piece of code may help you while populating the table. These details are returned from susi server which gets a list of all the registered in the following manner. First, it checks if base url of this user is something apart from admin. If not, it returns error which may look like this :

Failed to load resource: the server responded with a status of 401 (Base user role not sufficient. Your base user role is 'ANONYMOUS', your user role is 'anonymous')

Otherwise, it will generate a client identity, use to to get an authorization object which will loop through authorization.json file and return all the users encoded as JSONArray.

Additional Resources

  1. Official Material UI Documentation on Tables from marterial-ui
  2. Answer by Marco Bonelli on Stackoverflow on How to map JSON response in JavaScript?
  3. Answer by janpieter_z on Stackoverflow – on Render JSON data in ReactJS table

How to Send a Script as Variable to the Meilix ISO with Travis and Meilix Generator

We wanted to add more features to Melix Generator web app to be able to customize Meilix ISO with more features so we thought of sending every customization we want to apply as a different variable and then use the scripts from Meilix Generator repo to generate ISO but that idea was bad as many variables are to be made and need to be maintained on both Heroku and Travis CI and keep growing with addition of features to web app.

So we thought of a better idea of creating a combined script with web app for each feature to be applied to ISO and send it as a variable to Travis CI.

Now another problem was how to send script as a variable after generating it as json do not support special characters inside the script. We tried escaping the special characters and the data was successfully sent to Travis CI and was shown in config but when setting that variable as an environment variable in Travis CI the whole value of variable was not taken as we had spaces in the script.

So to eliminate that problem we encoded the variable in the app as base64 and sent it to Travis CI and used it using following code.

To generate the variable from script.

with open('','rb') as f:
    os.environ["TRAVIS_SCRIPT"] = str(base64.b64encode([1:]


For this we have to import base64 module and open the script generated in binary mode and using base64 we encode the script and using Travis CI API we send variable as script to the Travis CI to build the ISO with script in chroot we were also required to make changes in Meilix to be able to decode the script and then copy it into chroot during the ISO build.

sudo su <<EOF
echo "$TRAVIS_SCRIPT" > edit/
mv edit/


Using script inside chroot.

chmod +x
echo "$(<" #to test the file


Base64 python documentation from

Base64 bash tutorial from by Scott Miller

su in a script from answered by Ankit

SUSI.AI User Roles and How to Modify Them

In this blog, I discuss what is ‘user-role’ in SUSI.AI, what are the various roles and how SUSI admins can modify/update a user’s roles.

What is User Role?

A UserRole defines the servlet access right. Not all users are allowed to access all the data and services. For  example, To list all the users, minimal user role expected is ADMIN. This classification of users are inspired by the wikipedia User Access Levels, see querying SUSI, Users are classified into 7 different categories, namely :

  • BOT
  • USER  

* Please see that these are as of the date of publish of this blog. These are subject to change, which is very unlikely.

If SUSI is active as a bot on some bot integrated platform (like line or kik), the user role assigned to it will be that of BOT. This user role just has technical access to the server.

All the users who are not logged in but interacting with SUSI are ANONYMOUS users. These are only subject to chat, login and signup. They may use forgot password service and reset password services as well.

Once a user login to the server, a token is generated and sent back to client to maintain the identity, hence acknowledging them as USER(s).

Users with role assigned as “REVIEWERS” are expected to manage the Skill CMS. There might be some dispute or conflict in a skill. REVIEWERS then take the access of skill data and finalise the conflict there itself for smooth functionality.

ADMIN users are those who have special rights with them. These are more like moderators with much special rights than any other user.

At the top level of the hierarchy are the BUREAUCRATS. These users have more rights than anyone. They can change role of any other user, override decision of any ADMIN user as well. Both admins and bureaucrats have the access to all the settings file on the server. They not only can look at the list, but also download and upload them. Now these users also have right to upgrade or downgrade any other user as well.

All these user roles are defined in file.

In each request received by the server, the user role of user making the request is compared with the minimal user role in getMinimalUserRole() method. This method is defined in AbstractAPIHandler which validates if a user is allowed to access a particular servlet or not.

private void process(HttpServletRequest request, HttpServletResponse response, Query query) throws ServletException, IOException {
	// object initialisation and comparsions
// user authorization: we use the identification of the user to get the assigned authorization
        Authorization authorization = DAO.getAuthorization(identity);

        if (authorization.getUserRole().ordinal() < minimalUserRole.ordinal()) {
        	response.sendError(401, "Base user role not sufficient. Your base user role is '" + authorization.getUserRole().name() + "', your user role is '" + authorization.getUserRole().getName() + "'");
// evaluations based on other request parameters.

Now that we know about what User Roles actually are, let us look at how the servlet which allows the users {with at least ADMIN login} to change user role of some other user works.

In the request, 2 parameters are expected. These are :

  • user : email id of the user whose role has to be changed.
  • role : new role which will be assigned to this user.

Using a switch case, we identify the user role which is requested. If role is found to be null or any other value apart from “bot”, “anonymous”, “user”, “reviewer”, “accountcreator”, “admin” or “bureaucrat”, an error with error code 400 and error message “Bad User role” is thrown.

In the next steps, server generates client identity in order to get the corresponding Authorization object. If the user is not found in the database, again an error is thrown with error code 400 and error message “role not found

ClientCredential credential = new ClientCredential(ClientCredential.Type.passwd_login, userTobeUpgraded);
            ClientIdentity identity = new ClientIdentity(, credential.getName());
            if (!DAO.hasAuthorization(identity)) {
                throw new APIException(400, "Username not found");

By now, server is clear with the user identity and new role to be assigned. Since the user role is defined in authorization.json file, we overwrite the existing user role and finally server sends back the new user role of the use

Authorization auth = DAO.getAuthorization(identity);
            try {
            } catch (IllegalArgumentException e) {
                throw new APIException(400, "role not found");

            // Print Response
            result.put("newDetails", auth.getJSON());
            result.put("accepted", true);
            result.put("message", "User role changed successfully!!");
            return new ServiceResponse(result);


API to List All Users on SUSI.AI

In this blog, I discuss how the SUSI server helps in listing out all the users registered on it. The only role Susi server plays is, Whenever it receives a request at

The server evaluate the parameters in the request, validates them and notify the user accordingly. API needs 2 parameters, out of which access-token is a necessary. 2nd parameter has to be one from the given list :

Parameter Data type

  • getPageCount boolean
  • GetUserCount boolean
  • Page integer

On the basis of this 2nd parameter, server gets to know what does the client with given access-token is requesting. Server evaluates the access-token and validates that if the access token belongs to a user with user role atleast ADMIN, then the request is valid and proceed further with fetching the data in next step. Otherwise, server responds with error code “401” and error message “Base user role not sufficient”. It is advisable for clients that before redirecting users to admin panel or any other service, Please hit

And check that whether the user logged in is allowed to access the admin panel or not. The servlet /showAdminService.json is quite easy to understand for even those new to programming.

Coming back to our topic, by now, server knows that this client is authorized to access the user list. But what all information does server needs to provide? In response to this request, server encodes following attributes in the JSON Array {which is part of JSON object} and sends it to user :

Attribute Description

  • Name Email-Id of the user
  • Anonymous Is this user anonymous or not
  • User Role User Role of the user
  • Confirmed User has verified account or not
  • Last Login IP Last IP from which login was requested
  • Last Login Time Time when last login request was made
  • Signup Time When did the user signed up

First things first, check if enough parameters are provided or not. If not, respond with error stating “Bad Request. No parameter present”. Otherwise, server does a general iteration which has to be done irrespective of the 2nd parameter.

First of all, get a list of all the authorized users using getAuthorizedClients method of Data Access Object class. This method picks up all the keys from authorized file {which are nothing but identification of clients from which requests are received}. Though it, skips those key which are host addresses (which can not be used to identify a user), it does includes all the email ids {which are obvious identification of users}.

public static Collection<ClientIdentity> getAuthorizedClients() {
		ArrayList<ClientIdentity> i = new ArrayList<>();
		for (String id: authorization.keys()) {
			i.add(new ClientIdentity(id));
		return i;

In next steps, the collection is converted to suitable data types over which iterations are easy and can be converted to JSON objects and Arrays easily. After this, server evaluates which parameter is requested in the request. Let us pick each case one by one for simplicity.

  1. Client has requested number of pages in the request.

Server finds the size of keysArray {one of the object containing list of all the users}. Basic Mathematics to find out how many pages would be formed if size of each page is 50 elements and total elements are given.

if (call.get("getPageCount", false) == true) {
            int pageCount = keysArray.length % 50 == 0 ? (keysArray.length / 50) : (keysArray.length / 50) + 1;
            result.put("pageCount", pageCount);
            result.put("accepted", true);
            result.put("message", "Success: Fetched count of pages");
            return new ServiceResponse(result);
  1. User count is requested

Simply return sizeof list which has list of all the users. List to be used can be anyone from authorized, keysArray or any other derivative of authorized collection. Code is quite easy.

  1.      List of users on any page is requested.

Get the page number and after applying unitary maths, you will figure out the elementary of this.

for (Client client : authorized) {
                JSONObject json = client.toJSON();
                ClientIdentity identity = new ClientIdentity(, client.getName());
                Authorization authorization = DAO.getAuthorization(identity);
                UserRole userRole = authorization.getUserRole();
                json.put("userRole", userRole.toString().toLowerCase());

If any other attribute that is required, it’s encoding will be done here.  For example, to get user role of a user, generate a client identity followed by retrieval of user role from it. Encode it and send back to user.

Other details like last login IP, last login time and signup time are also fetched from respective files.


Change Password for SUSI Accounts Using Access Token and Email-ID

In this blog, I discuss how the SUSI server synchronizes with SUSI Accounts and SUSI webchat for users to Change Password. When a user logs in, the clients store the email id of the user along with the access token in cookies. These are stored once the client gets a positive login response from the server. Both of these are required at the time of making the final call. Web clients store the email id and access token in the following way.

cookies.set('loggedIn', loggedIn, { path: '/', maxAge: time });
cookies.set('emailId', email, { path: '/', maxAge: time });

First, the client has to ask the user to enter their current password. A javascript test is used to validate that at least 6 characters must be entered by the user. A similar test is run on the new password. But while confirming the password, client checks whether the user has entered the same password as new password or not. These are just the basics. In next stage (which is achieved only when all the above conditions are met), client encodes the email id (which it gets from cookies), current password, new password and the access token (which it again extracts from cookies).

Now, Client just has to make an ajax request to the server. The response is processed and sent back to the client. Let us now look at PasswordChange Servlet.

The base user role is defined as USER. Initial steps of the servlet are to extract the values form the request it receives. The values extracted from the request are in turn used to make a client’s identity. Before that, server checks if current and new password have same values or not. If not, then server returns a JSON response to user stating, “Your current password and new password matches”. Otherwise, it will continue its control flow as it is. Look at the code snippet below:

            result.put("message", "Your current password and new password matches");
            result.put("accepted", false);
            return new ServiceResponse(result);

The reader here may think that they have discovered a hack. But they have not. Why? Because this is just the first step. In later stages, the hash of passwords are used to match to see whether the passwords match or not. To obtain a proper client identity, first a Client credentials object is made with support from the email id which is received in ‘changepassword’ attribute. Using the ClientCredentials object made above, an object of Authentication class is made. This object uses a method defined in its class to return a valid client identity. Using the client identity, value of password hash is extracted from the database along with the salt used to hash the password. If any error is encountered while extracting the client’s password hash value and/or salt value, an error is thrown towards the client, with a message stating “invalid credentials”.

ClientCredential pwcredential = new ClientCredential(ClientCredential.Type.passwd_login, useremail);
            Authentication authentication = DAO.getAuthentication(pwcredential);
            ClientCredential emailcred = new ClientCredential(ClientCredential.Type.passwd_login,
            ClientIdentity identity = authentication.getIdentity();
            String passwordHash;
            String salt;

            try {
                passwordHash = authentication.getString("passwordHash");
                salt = authentication.getString("salt");
            } catch (Throwable e) {
                Log.getLog().info("Invalid password try for user: " + identity.getName() + " from host: " + post.getClientHost() + " : password or salt missing in database");
                result.put("message", "invalid credentials");
                throw new APIException(422, "Invalid credentials");

Using the same salt value that was used earlier, a hash for password entered by the user will be generated which now matches  the previous value. This is the point where the hack you were thinking you found, failed. Again the server throws an error message if user’s credential did not match. Passwords are hard to handle and easy to guess. So here we have used quite many tests before changing them. Users are not allowed to use their email id as a password as well.

If the server is clear on all the above facts and tests, It finally generates a new hashed value of the password received in the parameter ‘newpassword’ and replaces the old hash value with the new one. To notify the clients that password change exited with a success response, it sends a JSON object with message “Your password has been changed!” and accepted flag set to true.

if (DAO.hasAuthentication(emailcred)) {
                    Authentication emailauth = DAO.getAuthentication(emailcred);
                    String newsalt = createRandomString(20);
                    emailauth.put("passwordHash", getHash(newpassword, salt));
                    Log.getLog().info("password change for user: " + identity.getName() + " via newpassword from host: " + post.getClientHost());
                    result.put("message", "Your password has been changed!");
                    result.put("accepted", true);


Additional Resources:

Wikipedia article: What is DAO?

Backend Scraping in Loklak Server

Loklak Server is a peer-to-peer Distributed Scraping System. It scrapes data from websites and also maintain other sources like peers, storage and a backend server to scrape data. Maintaining different sources has it’s benefits of not engaging in costly requests to the websites, no scraping of data and no cleaning of data.

Loklak Server can maintain a secondary Loklak Server (or a backend server) tuned for storing large amount of data. This enables the primary Loklak Server fetch data in return of pushing all scraped data to the backend.

Lately there was a bug in backend search as a new feature of filtering tweets was added to scraping and indexing, but not for backend search. To fix this issue, I had backtracked the backend search codebase and fix it.

Let us discuss how Backend Search works:-

1) When query is made from search endpoint with:

a) source=all

When source is set to all. The first TwitterScraper and Messages from local search server is preferred. If the messages scraped are not enough or no output has been returned for a specific amount of time, then, backend search is initiated

b) source=backend

SearchServlet specifically scrapes directly from backend server.

2) Fetching data from Backend Server

The input parameters fetched from the client is feeded into DAO.searchBackend method. The list of backend servers fetched from config file. Now using these input parameters and backend servers, the required data is scraped and output to the client.

In DAO.searchOnOtherPeers method. the request is sent to multiple servers and they are arranged in order of better response rates. This method invokes method for sending request to the mentioned servers.

List<String> remote = getBackendPeers();
if (remote.size() > 0) {
    // condition deactivated because we need always at least one peer
    Timeline tt = searchOnOtherPeers(remote, q, filterList, order, count, timezoneOffset, where, SearchServlet.backend_hash, timeout);
    if (tt != null) tt.writeToIndex();
    return tt;


3) Creation of request url and sending requests

The request url is created according to the input parameters passed to method. Here the search url is created according to input parameters and request is sent to the respective servers to fetch the required messages.

   // URL creation
    urlstring = protocolhostportstub + "/api/search.json?q="
           + URLEncoder.encode(query.replace(' ', '+'), "UTF-8") + "&timezoneOffset="
           + timezoneOffset + "&maximumRecords=" + count + "&source="
           + (source == null ? "all" : source) + "&minified=true&shortlink=false&timeout="
           + timeout;
    if(!"".equals(filterString = String.join(", ", filterList))) {
       urlstring = urlstring + "&filter=" + filterString;
    // Download data
    byte[] jsonb = ClientConnection.downloadPeer(urlstring);
    if (jsonb == null || jsonb.length == 0) throw new IOException("empty content from " + protocolhostportstub);
    String jsons = UTF8.String(jsonb);
    JSONObject json = new JSONObject(jsons);
    if (json == null || json.length() == 0) return tl;
    // Final data fetched to be returned
    JSONArray statuses = json.getJSONArray("statuses");