DAO – blog.fossasia.org

API to List All Users on SUSI.AI

Post author:DravitLochan
Post published:August 27, 2017
Post category:FOSSASIA GSoC SUSI.AI
Post comments:0 Comments

In this blog, I discuss how the SUSI server helps in listing out all the users registered on it. The only role Susi server plays is, Whenever it receives a request at

http://api.susi.ai/aaa/getUsers.json

The server evaluate the parameters in the request, validates them and notify the user accordingly. API needs 2 parameters, out of which access-token is a necessary. 2nd parameter has to be one from the given list :

Parameter Data type

getPageCount boolean
GetUserCount boolean
Page integer

On the basis of this 2nd parameter, server gets to know what does the client with given access-token is requesting. Server evaluates the access-token and validates that if the access token belongs to a user with user role atleast ADMIN, then the request is valid and proceed further with fetching the data in next step. Otherwise, server responds with error code “401” and error message “Base user role not sufficient”. It is advisable for clients that before redirecting users to admin panel or any other service, Please hit

http://api.susi.ai/aaa/showAdminService.json

And check that whether the user logged in is allowed to access the admin panel or not. The servlet /showAdminService.json is quite easy to understand for even those new to programming.

Coming back to our topic, by now, server knows that this client is authorized to access the user list. But what all information does server needs to provide? In response to this request, server encodes following attributes in the JSON Array {which is part of JSON object} and sends it to user :

Attribute Description

Name Email-Id of the user
Anonymous Is this user anonymous or not
User Role User Role of the user
Confirmed User has verified account or not
Last Login IP Last IP from which login was requested
Last Login Time Time when last login request was made
Signup Time When did the user signed up

First things first, check if enough parameters are provided or not. If not, respond with error stating “Bad Request. No parameter present”. Otherwise, server does a general iteration which has to be done irrespective of the 2nd parameter.

First of all, get a list of all the authorized users using getAuthorizedClients method of Data Access Object class. This method picks up all the keys from authorized file {which are nothing but identification of clients from which requests are received}. Though it, skips those key which are host addresses (which can not be used to identify a user), it does includes all the email ids {which are obvious identification of users}.

public static Collection<ClientIdentity> getAuthorizedClients() {
		ArrayList<ClientIdentity> i = new ArrayList<>();
		for (String id: authorization.keys()) {
		    if(id.contains("host"))
		        continue;
			i.add(new ClientIdentity(id));
		}
		return i;
	}

In next steps, the collection is converted to suitable data types over which iterations are easy and can be converted to JSON objects and Arrays easily. After this, server evaluates which parameter is requested in the request. Let us pick each case one by one for simplicity.

Client has requested number of pages in the request.

Server finds the size of keysArray {one of the object containing list of all the users}. Basic Mathematics to find out how many pages would be formed if size of each page is 50 elements and total elements are given.

if (call.get("getPageCount", false) == true) {
            int pageCount = keysArray.length % 50 == 0 ? (keysArray.length / 50) : (keysArray.length / 50) + 1;
            result.put("pageCount", pageCount);
            result.put("accepted", true);
            result.put("message", "Success: Fetched count of pages");
            return new ServiceResponse(result);
        }

User count is requested

Simply return sizeof list which has list of all the users. List to be used can be anyone from authorized, keysArray or any other derivative of authorized collection. Code is quite easy.

List of users on any page is requested.

Get the page number and after applying unitary maths, you will figure out the elementary of this.

for (Client client : authorized) {
                JSONObject json = client.toJSON();
                ClientIdentity identity = new ClientIdentity(ClientIdentity.Type.email, client.getName());
                Authorization authorization = DAO.getAuthorization(identity);
                UserRole userRole = authorization.getUserRole();
                json.put("userRole", userRole.toString().toLowerCase());
                userList.add(json);
            }

If any other attribute that is required, it’s encoding will be done here. For example, to get user role of a user, generate a client identity followed by retrieval of user role from it. Encode it and send back to user.

Other details like last login IP, last login time and signup time are also fetched from respective files.

Resources

Post by Tutorials Point on How to get size of a collection.
Post by Wikipedia : What is DAO?

Implementation of SUSI.AI Signup Service

Post author:DravitLochan
Post published:August 27, 2017
Post category:API FOSSASIA GSoC SUSI.AI
Post comments:0 Comments

In this blog, I discuss the complete implementation of SUSI server when a user signs up with SUSI. Since public signup is enabled, minimal user role to access the servlet is ANONYMOUS. Given below is the API endpoint at which a signup request has to be made.

http://api.susi.ai/aaa/signup.json

Here we do not have much options unlike Login service. . Requests can be made in following ways with different parameters:

http://api.susi.ai/aaa/signup.json?signup=test@fossasia.com&password=Rest@123

http://api.susi.ai/aaa/signup.json?getParameters=true

http://api.susi.ai/aaa/signup.json?access_token={30 characters long access token}&validateEmail=test@fossasia.com&request_session=true

First one is the request which client makes, asking the server to register the user with email id test@fossasia.com and password as Reset@123. On receiving a request which has parameters signup and password, server looks back in the database and verifies that a user with same email id exists or not. If email id already exists, server throws an error stating “email already taken”. Given below is a small code snippet for better understanding.

ClientCredential credential = new ClientCredential(ClientCredential.Type.passwd_login, signup);
		Authentication authentication = DAO.getAuthentication(credential);

		if (authentication.getIdentity() != null) {
			throw new APIException(422, "email already taken");
		}

If client is not found in the database, a new identity is generated for the user and is set in authentication.json file. Next, a random salt string is generated which acts as salt to hash the user’s password. Since the same salt will be required at the time of matching the password for validation, we store both the salt and password hash in authentication.json file.

Now, if the database is going to encounter it’s first user, Automatically, BUREAUCRAT (Role with maximum power) will be assigned as his/her user role. To do this, we first get a list of all the users with the help of DAO class and count number of users present in it. Take a look at the code given below,

Authorization authorization = DAO.getAuthorization(identity);
		Collection<ClientIdentity> authorized = DAO.getAuthorizedClients();
		List<String> keysList = new ArrayList<String>();
		authorized.forEach(client -> keysList.add(client.toString()));
		String[] keysArray = keysList.toArray(new String[keysList.size()]);
		if(keysArray.length == 1) {
			authorization.setUserRole(UserRole.BUREAUCRAT);
		} else {
			authorization.setUserRole(UserRole.USER);
		}

In further steps, a random token is generated and marked as the verification token for the user who just signed-up. Other attributes for the token are also set like expiry time. Finally, account verification mail’s template is picked from “conf/templates/verification-mail” file and used as the body for the user verification email.

The second type of request is quite simple which is just to request a regular expression from the server to give proper information to the user. It’s code is quite easy and self understandable.

Third type of request is sent to the server when user clicks on the link received in the account verification email. Taking a closer look at the account verification link, one could easily decode that the request would be redirected to /aaa/signup.json with parameters

Access_token : 30 characters long access token
validateEmail : user’s email id
request_session : true

Access_token makes up the unique id for the user. ValidateEmail is to make it easier for server to generate a client identity. Let us now understand how server picks up each parameter and activates the user’s account.

if (post.get("validateEmail", null) != null) {
			if((auth.getIdentity().getName().equals(post.get("validateEmail", null)) && auth.getIdentity().isEmail()) // the user is logged in via an access token from the email {

				ClientCredential credential = new ClientCredential(ClientCredential.Type.passwd_login,
						auth.getIdentity().getName());
Authentication authentication =     DAO.getAuthentication(credential);

				if (authentication.getIdentity() == null) {
					authentication.delete();
					throw new APIException(400, "Bad request");
				}
				authentication.put("activated", true);
				result.put("accepted", true);
result.put("message", "You successfully verified your                                        account!");
				return new ServiceResponse(result);
			}
			throw new APIException(400, "Bad request"); // do not leak if user exists or not
		}

In the above code snippet, an object of ClientCredential class is generated and used to get authentication object for the user. If user is not found (name in authentication object is null), server throws error with error message “Bad Request”. Finally, If no error was encountered, server activates the user.

In getDefaultPermissions() function, permission for each user role is specified. All the users registered are allowed to register other users, but no user with anonymous user role can register anyone else except him/her.

Resources

Post by Tutorials Point on How to send email in JAVA
Post by Wikipedia on What is DAO?
Answer by Tomasz Nurkiewicz on StackoverFlow Need for server side account validation instead of client side

Data Access Layer in Open Event Organizer Android App

Post author:mejariamol
Post published:August 11, 2017
Post category:Android Event Event Management FOSSASIA GSoC Open Event
Post comments:0 Comments

Open Event Organizer is an Android App for Organizers and Entry Managers. Its core feature is scanning a QR Code to validate Attendee Check In. Other features of the App are to display an overview of sales and tickets management. The App maintains a local database and syncs it with the Open Event API Server. The Data Access Layer in the App is designed such that the data is fetched from the server or taken from the local database according to the user’s need. For example, simply showing the event sales overview to the user will fetch the data from the locally saved database. But when the user wants to see the latest data then the App need to fetch the data from the server to show it to the user and also update the locally saved data for future reference. I will be talking about the data access layer in the Open Event Organizer App in this blog.

The App uses RxJava to perform all the background tasks. So all the data access methods in the app return the Observables which is then subscribed in the presenter to get the data items. So according to the data request, the App has to create the Observable which will either load the data from the locally saved database or fetch the data from the API server. For this, the App has AbstractObservableBuilder class. This class gets to decide which Observable to return on a data request.

Relevant Code:

final class AbstractObservableBuilder<T> {
   ...
   ...
   @NonNull
   private Callable<Observable<T>> getReloadCallable() {
       return () -> {
           if (reload)
               return Observable.empty();
           else
               return diskObservable
                   .doOnNext(item -> Timber.d("Loaded %s From Disk on Thread %s",
                       item.getClass(), Thread.currentThread().getName()));
       };
   }

   @NonNull
   private Observable<T> getConnectionObservable() {
       if (utilModel.isConnected())
           return networkObservable
               .doOnNext(item -> Timber.d("Loaded %s From Network on Thread %s",
                   item.getClass(), Thread.currentThread().getName()));
       else
           return Observable.error(new Throwable(Constants.NO_NETWORK));
   }

   @NonNull
   private <V> ObservableTransformer<V, V> applySchedulers() {
       return observable -> observable
           .subscribeOn(Schedulers.io())
           .observeOn(AndroidSchedulers.mainThread());
   }

   @NonNull
   public Observable<T> build() {
       if (diskObservable == null || networkObservable == null)
           throw new IllegalStateException("Network or Disk observable not provided");

       return Observable
               .defer(getReloadCallable())
               .switchIfEmpty(getConnectionObservable())
               .compose(applySchedulers());
   }
}

The class is used to build the Abstract Observable which contains both types of Observables, making data request to the API server and the locally saved database. Take a look at the method build. Method getReloadCallable provides an observable which will be the default one to be subscribed which is a disk observable which means data is fetched from the locally saved database. The method checks parameter reload which if true suggests to make the data request to the API server or else to the locally saved database. If the reload is false which means data can be fetched from the locally saved database, getReloadCallable returns the disk observable and the data will be fetched from the locally saved database. If the reload is true which means data request must be made to the API server, then the method returns an empty observable.

The method getConnectionObservable returns a network observable which makes the data request to the API server. In the method build, switchIfEmpty operator is applied on the default observable which is empty if reload is true, and the network observable is passed to it. So when reload is true, network observable is subscribed and when it is false disk observable is subscribed. For example of usage of this class to make a events data request is:

public Observable<Event> getEvents(boolean reload) {
   Observable<Event> diskObservable = Observable.defer(() ->
       databaseRepository.getAllItems(Event.class)
   );

   Observable<Event> networkObservable = Observable.defer(() ->
       eventService.getEvents(JWTUtils.getIdentity(getAuthorization()))
           ...
           ...

   return new AbstractObservableBuilder<Event>(utilModel)
       .reload(reload)
       .withDiskObservable(diskObservable)
       .withNetworkObservable(networkObservable)
       .build();
}

So according to the boolean parameter reload, a correct observable is subscribed to complete the data request.

Links:
1. Documentation about the Operators in ReactiveX
2. Information about the Data Access Layer on Wikipedia

Using Data Access Object to Store Information

Post author:heysadboy
Post published:August 11, 2017
Post category:FOSSASIA Open Event
Post comments:0 Comments

We often need to store the information received from the network to retrieve that later. Although we can store and read data directly but by using data access object to store information enables us to do data operations without exposing details of the database. Using data access object is also a best practice in software engineering. Recently I modified Connfa app to store the data received in Open Event format. In this blog, I describe how to use data access object.

The goal is to abstract and encapsulate all access to the data and provide an interface. This is called Data Access Object pattern. In a nutshell, the DAO “knows” which data source (that could be a database, a flat file or even a WebService) to connect to and is specific for this data source. It makes no difference in applications when it accesses a relational database or parses xml files (using a DAO). The DAO is usually able to create an instance of a data object (“to read data”) and also to persist data (“to save data”) to the datasource.

Consider the example from Connfa app in which get the tracks from API and store them in SQL database. We use DAO to create a layer between model and database. Where AbstractEntityDAO is an abstract class which have the functions to perform CRUD operation. We extend it to implement them in our DAO model. Here is TrackDAO structure,

public class TrackDao extends AbstractEntityDAO<Track, Long> {

    public static final String TABLE_NAME = "table_track";

    @Override
    protected String getSearchCondition() {
        return "_id=?";
    }
    
    ...
}

Find the complete class to see the detailed methods to implement search conditions, get key columns, create instance etc. here.

Here is a general method to get the data from the database. Where getFacade() for the given layer element, this method returns the requested facade object to represent the passed in layer element.

public List<ClassToSave> getAllSafe() {
   ILAPIDBFacade facade = getFacade();
   try {
       facade.open();
       return getAll();

   } finally {
       facade.close();
   }
}

Now we can create an instance to use these methods instead of directly using SQL operations. This functions gets the data and sort it accordingly.

private TrackDao mTrackDao;
 public List<Track> getTracks() {
   List<Track> tracks = mTrackDao.getAllSafe();
   Collections.sort(tracks, new Comparator<Track>() {
       @Override
       public int compare(Track track, Track track2) {
           return Double.compare(track.getOrder(), track2.getOrder());
       }
   });
   return tracks;
}

References:

Wikipedia – https://en.wikipedia.org/wiki/Data_access_object
Sun Microsystems – http://www.oracle.com/technetwork/java/dataaccessobject-138824.html

Download SUSI.AI Setting Files from Data Folder

Post author:DravitLochan
Post published:July 23, 2017
Post category:API FOSSASIA GSoC Open Event SUSI.AI Tutorial
Post comments:0 Comments

In this blog, I will discuss how the DownloadDataSettings servlet hosted on SUSI server functions. This post also covers a step by step demonstration on how to use this feature if you have hosted your own custom SUSI server and have admin rights to it. Given below is the endpoint where the request to download a particular file has to be made.

/data/settings

For systematic functionality and workflow, Users with admin login, are given a special access. This allows them to download the settings files and go through them easily when needed. There are various files which have email ids of registered users (accounting.json), user roles associated to them (authorization.json), groups they are a part of (groups.json) etc. To list all the files in the folder, use the given below end point:

/aaa/listSettings.json

How does the above servlet works? Prior to that, let us see how to to get admin rights on your custom SUSI.AI server.
For admin login, it is required that you have access to files and folders on server. Signup with an account and browse to

/data/settings/authorization.json

Find the email id with which you signed up for admin login and change userRole to “admin”. For example,

{
	"email:test@test.com": {
		"permissions": {},
		"userRole": "user"
	}
}

If you have signed up with an email id “test@test.com” and want to give admin access to it, modify the userRole to “admin”. See below.

{
	"email:test@test.com": {
		"permissions": {},
		"userRole": "admin"
	}
}

Till now, server did not have any email id with admin login or user role equal to admin. Hence, this exercise is required only for the first admin. Later admins can use changeUserRole application and give/change/modify user roles for any of the users registered. By now you must have admin login session. Let’s see now how the download and file listing servlets work.
First, the server creates a path by locally referencing settings folder with the help of DAO.data_dir.getPath(). This will give a string path to the data directory containing all the data-settings files. Now the server just has to make a JSONArray and has to pass a String array to JSONArray’s constructor, which will eventually be containing the name of all the data/settings files. If the process is not successfull ,then, “accepted” = false will be sent as an error to the user. The base user role to access the servlet is ADMIN as only admins are allowed to download data/setting files,
The file name which you have to download has to be sent in a HTTP request as a get parameter. For example, if an admin has to download accounting.json to get the list of all the registered users, the request is to be made in the following way:

BASE_URL+/data/settings?file=file_name

*BASE_URL is the URL where the server is hosted. For standard server, use BASE_URL = http://api.susi.ai.

In the initial steps, Server generates a path to data/settings folder and finds the file, name of which it receives in the request. If no filename is specified in the API call, by default, the server sends accounting.json file.

File settings = new File(DAO.data_dir.getPath()+"/settings");
String filePath = settings.getPath(); 
String fileName = post.get("file","accounting"); 
filePath += "/"+fileName+".json";

Next, the server will extract the file and using ServletOutputStream class, it will generate properties for it and set appropriate context for it. This context will, in turn, fetch the mime type for the file generated. If the mime type is returned as null, by default, mime type for the file will be set to application/octet-stream. For more information on mime type, please look at the following link. A complete list of mime types is compiled and documented here.

response.setContentType(mimetype);
response.setContentLength((int)file.length());

In the above code snippet, mime type and length of the file being downloaded is set. Next, we set the headers for the download response and use filename for that.

response.setHeader("Content-Disposition", "attachment; filename=" + fileName +".json");

All the manual work is done by now. The only thing left is to open a buffer stream, size of which has been defined as a class variable.
Here we use a byte array of size 4096 elements and write the file to client’s default download storage.

private static final int BUFSIZE = 4096;
byte[] byteBuffer = new byte[BUFSIZE];
             DataInputStream in = new DataInputStream(new FileInputStream(file));
            while ((in != null) && ((length = in.read(byteBuffer)) != -1))
            {
                outStream.write(byteBuffer,0,length);
            }

            in.close();
            outStream.close();

All the above-mentioned steps are enclosed in a try-catch block, which catches an exception if any ,and logs it in the log file. This message is also sent to the client for appropriate user information along with the success or failure indication through a boolean flag. Do not forget to close the input and output buffers as it may lead to memory leaks and someone with proper knowledge of network and buffer stream would be able to steal any essential or secured data.

Additional Resources

Complete list of mime types -Post by developer.mozilla (public blog post)
What are mime types -Stackoverflow (answer by Quentin)
How to download a file from java servlet -Post by javatpoint
Request headers explained -Post by w3.org