Implementing Loklak APIs in Java using Reflections

Loklak server provides a large API to play with the data scraped by it. Methods in java can be implemented to use these API endpoints. A common approach of implementing the methods for using API endpoints is to create the request URL by taking the values passed to the method, and then send GET/POST request. Creating the request URL in every method can be tiresome and in the long run maintaining the library if implemented this way will require a lot of effort. For example, assume a method is to be implemented for suggest API endpoint, which has many parameters, for creating request URL a lot of conditionals needs to be written – whether a parameter is provided or not.

Well, the methods to call API endpoints can be implemented with lesser and easy to maintain code using Reflection in Java. The post ahead elaborates the problem, the approach to solve the problem and finally solution which is implemented in loklak_jlib_api.

Let’s say, the status API endpoint needs to be implemented, a simple approach can be:

public class LoklakAPI {
    
    public static String status(String baseUrl) {
        String requestUrl = baseUrl   "/api/status.json";
        
        // GET request using requestUrl 
    }

    public static void main(String[] argv) {
        JSONObject result = status("https://api.loklak.org");
    }
}

This one is easy, isn’t it, as status API endpoint requires no parameters. But just imagine if a method implements an API endpoint that has a lot of parameters, and most of them are optional parameters. As a developer, you would like to provide methods that cover all the parameters of the API endpoint. For example, how a method would look like if it implements suggest API endpoint, the old SuggestClient implementation in loklak_jlib_api does that:

public static ResultList<QueryEntry> suggest(
           final String hostServerUrl,
           final String query,
           final String source,
           final int count,
           final String order,
           final String orderBy,
           final int timezoneOffset,
           final String since,
           final String until,
           final String selectBy,
           final int random) throws JSONException, IOException {
       ResultList<QueryEntry>  resultList = new ResultList<>();
       String suggestApiUrl = hostServerUrl
               + SUGGEST_API
               + URLEncoder.encode(query.replace(' ', '+'), ENCODING)
               + PARAM_TIMEZONE_OFFSET + timezoneOffset
               + PARAM_COUNT + count
               + PARAM_SOURCE + (source == null ? PARAM_SOURCE_VALUE : source)
               + (order == null ? "" : (PARAM_ORDER + order))
               + (orderBy == null ? "" : (PARAM_ORDER_BY + orderBy))
               + (since == null ? "" : (PARAM_SINCE + since))
               + (until == null ? "" : (PARAM_UNTIL + until))
               + (selectBy == null ? "" : (PARAM_SELECT_BY + selectBy))
               + (random < 0 ? "" : (PARAM_RANDOM + random))
               + PARAM_MINIFIED + PARAM_MINIFIED_VALUE;
 
                // GET request using suggestApiUrl
   }
}

A lot of conditionals!!! The targeted users may also get irritated if they need to provide all the parameters every time even if they don’t need them. The obvious solution to that is overloading the methods. But,  then again for each overloaded method, the same repetitive conditionals need to be written, a form of code duplication!! And what if you have to implement some 30 API endpoints and in future maintaining them, a single thought of it is scary and a nightmare for the developers.

Approach to the problem

To reduce the code size, one obvious thought that comes is to implement static methods that send GET/POST requests provided request URL and the required data. These methods are implemented in loklak_jlib_api by JsonIO and NetworkIO.

You might have noticed the method name i.e. status, suggest is also the part of the request URL. So, the common pattern that we get is:

base_url + "/api/" + methodName + ".json" + "?" + firstParameterName + "=" + firstParameterValue + "&" + secondParameterName + "=" + secondParameterValue ...

Reflection API to rescue

Using Reflection API inspection of classes, interfaces, methods, and variables can be done, yes you guessed it right if we can inspect we can get the method and parameter names. It is also possible to implement interfaces and create objects at runtime.

So, an interface is created and API endpoint methods are defined in it. Using reflection API the interface is implemented lazily at runtime. LoklakAPI interface defines all the API endpoint methods. Some of the defined methods are:

public interface LoklakAPI {
 
    @Target(ElementType.METHOD)
    @Retention(RetentionPolicy.RUNTIME)
    @interface GET {}
 
    @Target(ElementType.METHOD)
    @Retention(RetentionPolicy.RUNTIME)
    @interface POST {}
 
    @GET
    JSONObject search(String q);
 
    @GET
    JSONObject search(String q, int count);
 
    @GET
    JSONObject search(String q, int timezoneOffset, String since, String until);
 
    @GET
    JSONObject search(String q, int timezoneOffset, String since, String until, int count);
 
    @GET
    JSONObject peers();
 
    @GET
    JSONObject hello();
 
    @GET
    JSONObject status();
 
    @POST
    JSONObject push(JSONObject data);
}

Here GET and POST annotations are used to mark the methods which use GET and POST request respectively, so that appropriate method for GET and POST static method can be used JsonIOloadJson for GET request and pushJson for POST request.

A private static inner class ApiInvocationHandler is created in APIGenerator, which implements InvocationHandler – used for implementing interfaces at runtime.

private static class ApiInvocationHandler implements InvocationHandler {
 
   private String mBaseUrl;
 
   public ApiInvocationHandler(String baseUrl) {
       this.mBaseUrl = baseUrl;
   }
 
   @Override
   public Object invoke(Object o, Method method, Object[] values) throws Throwable {
       Parameter[] params = method.getParameters();
       Object[] paramValues = values;
 
       /*
       format of annotation name:
       @packageWhereAnnotationIsDeclared$AnnotationName()
       Example: @org.loklak.client.LoklakAPI$GET()
       */
       Annotation annotation = method.getAnnotations()[0];
       String annotationName = annotation.toString().toLowerCase();
 
       String apiUrl = createGetRequestUrl(mBaseUrl, method.getName(),
               params, paramValues);
       if (annotationName.contains("get")) { // GET REQUEST
           return loadJson(apiUrl);
       } else { // POST REQUEST
           JSONObject jsonObjectToPush = (JSONObject) paramValues[0];
           String postRequestUrl = createPostRequestUrl(mBaseUrl, method.getName());
           return pushJson(postRequestUrl, params[0].getName(), jsonObjectToPush);
       }
   }
}

The base URL is provided while creating the object, in the constructor. The invoke method is called whenever a defined method in the interface is called in actual code. This way an interface is implemented at runtime. The parameters of invoke method:

  • Object o – the object on which to call the method.
  • Method method – method which is called, parameters and annotations of the method are obtained using getParameters and getAnnotations methods respectively which are required to create our request url.
  • Object[] values – an array of parameter values which are passed to the method when calling it.

createGetRequestUrl is used to create the request URL for sending GET requests.

private static String createGetRequestUrl(
       String baseUrl, String methodName, Parameter[] params, Object[] paramValues) {
   String apiEndpointUrl = baseUrl.replaceAll("/+$", "") + "/api/" + methodName + ".json";
   StringBuilder url = new StringBuilder(apiEndpointUrl);
   if (params.length > 0) {
       String queryParamAndVal = "?" + params[0].getName() + "=" + paramValues[0];
       url.append(queryParamAndVal);
       for (int i = 1; i < params.length; i++) {
           String paramAndVal = "&" + params[i].getName()
                   + "=" + String.valueOf(paramValues[i]);
           url.append(paramAndVal);
       }
   }
   return url.toString();
}

Similarly, createPostRequestUrl is used to create request URL for sending POST requests.

private static String createPostRequestUrl(String baseUrl, String methodName) {
   baseUrl = baseUrl.replaceAll("/+$", "");
   return baseUrl + "/api/" + methodName + ".json";
}

Finally, Proxy.newProxyInstance is used to implement the interface at runtime. An instance of ApiInvocationHandler class is passed to the newProxyInstance method. This is all done by static method createApiMethods.

public static <T> T createApiMethods(Class<T> service, final String baseUrl) {
   ApiInvocationHandler apiInvocationHandler = new ApiInvocationHandler(baseUrl);
   return (T) Proxy.newProxyInstance(
           service.getClassLoader(), new Class<?>[]{service}, apiInvocationHandler);
}

Where:

  • Class<T> service – is the interface where API endpoint methods are defined, here LoklakAPI.class.
  • String baseUrl – web address of the server where Loklak server is hosted.

NOTE: For all this to work the library must be build using “-parameters” flag, else parameter names can’t be obtained. Since loklak_jlib_api uses maven, “-parameters” is provided in pom.xml file.

A small example:

String baseUrl = "https://api.loklak.org";
LoklakAPI loklakAPI = APIGenerator.createApiMethods(LoklakAPI.class, baseUrl);
 
JSONObject searchResult = loklakAPI.search("FOSSAsia");
Continue ReadingImplementing Loklak APIs in Java using Reflections

Adding a new Servlet/API to SUSI Server for Skill Wiki

Susi skill wiki is an editor to write and edit skill easily. It follows an API-centric approach where the Susi server acts as API server and a web front-end  act as the client for the API and provides the user interface. A skill is a set of intents. One text file represents one skill, it may contain several intents which all belong together.

The schema for storing a skill is as following:

Using this, one can access any skill based on four tuples parameters model, group, language, skill.  To achieve this on server side let’s create an API endpoint to list all skills based on given model, groups and languages. To check the source for this endpoint clone the susi_server repository from here.

git clone https://github.com/fossasia/susi_server.git

Have a look at documentation for more information about Susi Server.

The Servlet java file is placed in susi_server/ai/susi/server/api/cms/ListSkillService. To implement the endpoint we will use the HttpServlet class which provides methods, such as doGet and doPost, for handling HTTP-specific services.

In Susi Server an abstract class AbstractAPIHandler extending HttpServelets and implementing API handler interface is provided.  Next we will inherit our ListSkillService class from AbstractAPIHandler and implement APIhandler interface.

To implement our servlet we will be overriding 4 methods namely

  • Minimal Base User role 

public BaseUserRole getMinimalBaseUserRole() {
return BaseUserRole.ANONYMOUS;
}

This method tells the minimum Userrole required to access this servlet it can also be ADMIN, USER. In our case it is Anonymous. A User need not to log in to access this endpoint.

  • Default Permissions  

public JSONObject getDefaultPermissions(BaseUserRole baseUserRole) {
return null;
}

This method returns the default permission attached with base user role, our servlets has nothing to do with it, therefore we can simply return null for this case.

  • The API Path 

public String getAPIPath() {
return "/cms/getSkillList.json";
}

This methods sets the API endpoint path, it gets appended to base path which is 127.0.0.1:4000/cms/getSkillList.json for the local host and http://api.susi.ai/cms/getSkillList.json for the server.

  • The ServiceImpl method 

public ServiceResponse serviceImpl(Query call, HttpServletResponse response, Authorization rights, final JsonObjectWithDefault permissions) {

        String model_name = call.get("model", "general");
        File model = new File(DAO.model_watch_dir, model_name);
        String group_name = call.get("group", "knowledge");
        File group = new File(model, group_name);
        String language_name = call.get("language", "en");
        File language = new File(group, language_name);

        ArrayList fileList = new ArrayList();
        fileList =  listFilesForFolder(language, fileList);
        JSONArray jsArray = new JSONArray(fileList);

        JSONObject json = new JSONObject(true)
                .put("model", model_name)
                .put("group", group_name)
                .put("language", language_name)
                .put("skills", jsArray);
        return new ServiceResponse(json);

    }

    ArrayList listFilesForFolder(final File folder,  ArrayList fileList) {

        File[] filesInFolder = folder.listFiles();
        if (filesInFolder != null) {
            for (final File fileEntry : filesInFolder) {
                if (!fileEntry.isDirectory()) {
                    fileList.add(fileEntry.getName()+"");
                }
            }
        }
        return  fileList;
    }

To access any skill we need parameters model, group, language. We get this through call.get method where first parameter is the key for which we want to get the value and second parameter is the default value. Based on received model, group and language browse files in that folder and put them in Json array to return the Service Json response.
That’s all you have successfully implemented the API endpoint to list all the skills in given model group and language. Move on and test it.

And see the resultsTo access any skill we need  parameters model, group, language. We get this through call.get method where first parameter is the key for which we want to get the value and second parameter is the default value. Based on received model, group and language browse files in that folder and put them in Json array to return the Service Json response.

 You have  successfully implemented the API endpoint to list all the skills in given model group and language. Move on and test it.

It’s easy isn’t it you have learnt how to add an endpoint in the server, it’s time to go ahead and create more endpoints to enhance Susi Server, take a look and contribute to Susi Server.

Continue ReadingAdding a new Servlet/API to SUSI Server for Skill Wiki

Implementing Voice Search In Susper (in Chrome only)

Last week @mariobehling opened up an issue to implement voice search in Susper. Google Chrome provides an API to integrate Speech recognition feature with any website. More about API can be read here: https://shapeshed.com/html5-speech-recognition-api/

The explanation might be in Javascript but it has been written following syntax of Angular 4 and Typescript. So, I created a speech-service including files:

  • speech-service.ts
  • speech-service.spec.ts

Code for speech-service.ts: This is the code which will control the working of voice search.

import { Injectable, NgZone } from ‘@angular/core’;
import { Observable } from ‘rxjs/Rx’;
interface
IWindow extends Window {
  webkitSpeechRecognition: any;
}
@Injectable()
export
class SpeechService {

constructor(private zone: NgZone) { }

record(lang: string): Observable<string> {
  return Observable.create(observe => {
    const { webkitSpeechRecognition }: IWindow = <IWindow>window;

    const recognition = new webkitSpeechRecognition();

    recognition.continuous = true;
    recognition.interimResults = true;
    recognition.onresult = take => this.zone.run(() => observe.next(take.results.item(take.results.length 1).item(0).transcript)
);

    recognition.onerror = err =>observe.error(err);
    recognition.onend = () => observe.complete();
    recognition.lang = lang;
    recognition.start();
});
}
}

You can find more details about API following the link which I have provided above in starting. Here recognition.onend() => observe.complete() works as an important role here. Many developers forget to use it when working on voice search feature. It works like: whenever a user stops speaking, it will automatically understand that voice action has now been completed and the search can be attempted. And for this:

speechRecognition() {
  this.speech.record(‘en_US’).subscribe(voice => this.onquery(voice));
}

We have used speechRecognition() function. onquery() function is called when a query is entered in a search bar.

Default language has been set up as ‘en_US’ i.e English. We also created an interface to link it with the API which Google Chrome provides for adding voice search feature on any website.

I have also used a separate module by name NgZone. Now, what is NgZone? It is used as an injectable service for executing working inside or outside of the Angular zone. I won’t go into detail about this module much here. More about it can be found on angular-docs website.

We have also, implemented a microphone icon on search bar similar to Google. This is how Susper’s homepage looks like now:

This feature only works in Google Chrome browser and for Firefox it doesn’t. So, for Firefox browser there was no need to show ‘microphone’ icon since voice search does not work Firefox. What we did simply use CSS code like this:

@mozdocument urlprefix() {
  .microphone {
    display: none;
  }
}

@-moz-document url-prefix() is used to target elements for Firefox browser only. Hence using, this feature we made it possible to hide microphone icon from Firefox and make it appear in Chrome.

For first time users: To use voice search feature click on the microphone feature which will trigger speechRecognition() function and will ask you permission to allow your laptop/desktop microphone to detect your voice. Once allowing it, we’re done! Now the user can easily, use voice search feature on Susper to search for a random thing.

Continue ReadingImplementing Voice Search In Susper (in Chrome only)

How to add a new Servlet/API to SUSI Server

You have got a new feature added to enhance SUSI-AI (in web/android/iOS application) but do not find an API which could assist you in your work to make calls to the server {since the principle of all Susi-AI clients is to contact with SUSI-server for any feature}. Making servlets for  Susi is quite different from a normal JAVA servlet. Though the real working logic remains the same but we have got classes which allow you to directly focus on one thing and that is to maintain your flow for the feature. To find already implemented servlets, first clone the susi_server repository  from here.

git clone https://github.com/fossasia/susi_server.git

Cd to susi_server directory or open your terminal in susi_server directory. (This blog focuses on servlet development for Susi only and hence it is assumed that you have any version of JAVA8 installed properly). If you have not gone through how to run a susi_server manually, then follow  below steps to start the server:

./gradlew build	   //some set of files and dependencies will be downloaded
bin/start.sh		   //command to start the server

This will start your Susi server and it will listen at port 4000.

The first step is to analyze that to which class of API is your  servlet  going to be added. Let us take a small example and see how to proceed step by step. Let us look at development of ListSettingsService servlet. (to find the code of this servlet, browse to the following location: susi_server->src->ai->susi->server->api->aaa). Once you have decided the classification of your srvlet, create a .java file in it (Like we created ListSettingsService.java file in aaa folder). Extend AbstractAPIHandler class to your class and implement APIHandler to your class. If you are using any IDE like Intelij IDEA or eclipse then they will give you an error message and when you click on it, it  will ask you to Override some methods. Select the option and if you are using a simple text editor, then override the following methods in the given way:

@Override
    public String getAPIPath() {
        return null;
    }
@Override
    public BaseUserRole getMinimalBaseUserRole() {
        return null;
    }
@Override
    public JSONObject getDefaultPermissions(BaseUserRole baseUserRole) {
        return null;
    }
@Override
    public ServiceResponse serviceImpl(Query post, HttpServletResponse response, Authorization rights, JsonObjectWithDefault permissions) throws APIException {
        Return null;
    }

What all these methods are for and why do we need them?

These are those 4 methods that make our work way easy. With the code compilation, first getAPIPath() is called to evaluate the end point.  Whenever this end point is called properly, it responds with whatever is defined in serviceImpl(). In our case we have given the endpoint

"/aaa/listSettings.json".

Ensure that you do not have 2  servlets with same end point.

Next in the line is getMinimalBaseUserRole() method. While developing certain features, a need of special privilege {like admin login} might be required. If you are implementing a feature for Admins only (like  we are doing in this servlet), return BaseUserRole.ADMIN. If you want to give access to anyone (registered or not) then return BaseUserRole.Anonymous. These might be login, signup or maybe a search point. By default all these methods are returning null. Once you are decided what to return, encode it in serviceImpl() method.

Look at the below implementation of the servlet :

@Override
    public String getAPIPath() {
        return "/aaa/listSettings.json";
}

@Override
    public BaseUserRole getMinimalBaseUserRole() {
        return BaseUserRole.ADMIN;
}

@Override
    public JSONObject getDefaultPermissions(BaseUserRole baseUserRole) {
        return null;
}

@Override
    public ServiceResponse serviceImpl(Query post, HttpServletResponse response, Authorization rights, JsonObjectWithDefault permissions) throws APIException {

        String path = DAO.data_dir.getPath()+"/settings/";
        File settings = new File(path);
        String[] files = settings.list();
        JSONArray fileArray = new JSONArray(files);
        return new ServiceResponse(fileArray);
    }
}

As discussed earlier, the task of this servlet is to list all the files in data/settings folder. But this list is only available to users with admin login.

DAO.data_dir.getPath() returns a String identifier which is the path to data directory present in susi_server folder. We append “/settings/” to access settings folder inside it. Next we list all the files present in settings folder, encode them as a JSONArray object and reeturn the JSONArray object.

Think you can enhance Susi-server now? Get started right away!!

Continue ReadingHow to add a new Servlet/API to SUSI Server

Integrating an Image Editing Page in Phimpme Android

The main aim of the Phimpme is to develop image editing and sharing application as an alternative to proprietary solutions like Instagram. Any user can choose a photo from the gallery or click a picture from the camera and upload it on the various social media platform including Drupal and wordpress. As most of the image editor applications in the app store currently my team and I discussed and listed down the basic functionality of the Image editing activity. We have listed down the following features for image Editing activity:

  • Filters.
  • Stickers
  • Image tuning

Choosing the Image Editing Application

There are number of existing Open Source projects that we went through to check how they could be integrated into Phimpme. We looked into those projects which are licensed under the  MIT Licence. As per the MIT Licence the user has the liberty to modify the use the code, modify it, merge, publish it without any restrictions. Image-Editor Android is one such application which has MIT Licence. The Image-Editor Android has extensive features for manipulating and enhancing the image. The features are as follows:

  • Edit Image by drawing on it.
  • Applying stickers on the image.
  • Applying filters.
  • Crop.
  • Rotating the image.
  • Text on the image.

It is an ideal application to be implemented in our project.

The basic flow of the application

First, getting the image either by gallery or camera. The team has implemented leafPic and openCamera. Second, redirecting the image from the leafPic gallery to the Image editing activity by choosing edit option from the popup menu.

Populating the Menu in the popup menu in XML:

<menu> tag is the root node, which contains ites in the popup menu. The following code is used to populate the menu:

<?xml version="1.0" encoding="utf-8"?>
<menu xmlns:android="http://schemas.android.com/apk/res/android">
    <item android:id="@+id/action_edit"
          android:icon="@drawable/ic_edit"
          android:title="@string/Edit"
          android:showAsAction="ifRoom"/>
    <item android:id="@+id/action_use_as"
          android:icon="@drawable/ic_use_as"
          android:title="@string/useAs" />
</menu>

Setting up the Image Editing Activity

Image-Editor Android application contains two main sections.

  • MainActivity (To get the image).
  • imageeditlibrary(To edit the image)

We need to import imageeditlibrary module. Android studios gives easy method to import a module from any other project using GUI. This can be done as follows: File->new->import module then choosing the module from the desired application.

Things to remember after importing any module from any other project:

  • Making sure that the minSdkVersion and targetSdkVersion in the gradle of the imported module and the current working project is same. In Phimpme the minSdkVersion is 16 and tagetSdkVersion is 25, which is used as standard SDK version.
  • Importing all the classes in the used in the imageeditlibrary module before using them in the leadPic gallery.

Sending Image to Image Editing Activity

This includes three tasks:

  • Handling onclick listeners.
  • Sending the image from the leafPic Activity
  • Receiving the the image in EditImageActivity.

Handling onClick Listener:

public boolean onOptionsItemSelected(MenuItem item) {
        switch (item.getItemId()) {
case R.id.action_edit:
// setOnclick listener here.
  }
}

Sending Image to EditImageActivity:

First we need to get the path of the image to be send. For this we need FileUtils class to handle the source address of the image. In FileUtils we use the function getEditFile().

public static File genEditFile(){
        return FileUtils.getEmptyFile("croppedImage"
                + System.currentTimeMillis() + ".png");
    }

Which calls the function getEmptyFile(String name):

public static File getEmptyFile(String name) {
        File folder = FileUtils.createFolders();
        if (folder != null) {
            if (folder.exists()) {
                File file = new File(folder, name);
                return file;
            }
        }
        return null;
    }

After getting the path of the file we need to send the path of the file to the EditImageActivity:

Uri uri = Uri.fromFile(new File(getAlbum().getCurrentMedia().getPath()));
                File outputFile = FileUtils.genEditFile();
                EditImageActivity.start(this,String.valueOf(uri),outputFile.getAbsolutePath(),ACTION_REQUEST_EDITIMAGE);

Receiving the image in EdtiImageActivity:

This is done by calling getdata() function in onCreate function.

private void getData() {
        filePath = getIntent().getStringExtra(FILE_PATH);
        saveFilePath = getIntent().getStringExtra(EXTRA_OUTPUT);
        loadImage(filePath);
    }

EditImageActivity Layout:

Conclusion

When integrating files from another activity we have to keep the API version of both the projects same. The best way to send an image to another activity is to save the image internally and then call the image path from the other activity.

Continue ReadingIntegrating an Image Editing Page in Phimpme Android

Implementing a chatbot using the SUSI.AI API

SUSI AI is an intelligent Open Source personal assistant. It is a server application which is able to interact with humans as a personal assistant. The first step in implementing a bot using SUSI AI is to specify the pathway for query response from SUSI AI server.

The steps mentioned below provide a step-by-step guide to establish communication with SUSI AI server:

    1. Given below is HTML code that demonstrates how to connect with SUSI API through an AJAX call. To put this file on a Node Js server, see Step 2.  To view the response of this call, follow Step 4.

      <!DOCTYPE html>
      <body>
      <h1>My Header</h1>
      <p>My paragraph.</p>
      //Script with source here 
      //Script to be written here 
      </body>
      </html>
      

      In above code add scripts given below and end each script with closing tag </script>. In the second script we are calling SUSI API with hello query and showing data that we are receiving through call on console.

      <script src="https://ajax.googleapis.com/ajax/libs/jquery/3.2.1/jquery.min.js">
    2. <script>
      $(function (){ 
          $.ajax({ 
              dataType: 'jsonp', 
              type:'GET', url: 'http://api.susi.ai/susi/chat.json? timezoneOffset=-300&q=hello', 
               success: function(data){ 
                   console.log('success', data); 
               } 
          }); 
      });
    3. Code below is in node js to setup localhost and getting the same above result on browser. Below is Node Js code to setup a server at localhost for the above created HTML file.

      var http = require('http');
       var fs = require('fs');
       http.createServer(function (req, res) {
        fs.readFile('YOURFILENAME.html', function(err, data) {
          res.writeHead(200, {'Content-Type': 'text/html'});
          res.write(data);
          res.end();
        });
       }).listen(9000);
    4. We will get following response by running this Node js code and checking results on http://localhost:9000/ To run this code install Node Js and write “node filename.js” in command line.
    5. You can open above window by right clicking on page and selecting Inspect. Go to the Network option and select the relevant api call from left section of Inspect window.

We have successfully got response from SUSI API and now we can use this response for building bots for receiving replies for user.

Continue ReadingImplementing a chatbot using the SUSI.AI API

Generating the Google IO Open Event Android App

The main aim of FOSSASIA Open Event Android App is to give an event organiser the ability to generate the app through a single click by providing the necessary json and binary files. As of late the Android application was tested on Google IO 2017 event. The sample files can be seen here. The data with respect to the event was taken from this site (https://events.google.com/io/). What was astonishing about this application is the simplicity with which we can make an event specific application by giving the vital assets required (json and binary files).

What was needed for generating the Google IO 2017 app?

For generating the app we had to provide the following files:

  1. images folder containing the necessary images of speaker, the logo of the event etc.
  2. event json file which has all the event specific information like the name of the event, the schedule of the event, the description of the event etc.
  3. forms json file having session and speaker form data.
  4. meta json file having the root url of the event.
  5. microlocations json file having all the locations where the events are going to happen.
  6. session_types json file consisting data of all the type of session which will occur in the vent.
  7. sessions json file consisting session specific data like the title of the session, start time and end time of session, which track that session belongs to etc.
  8. speakers json file consisting of speaker specific data like the name of the speaker, image of the speaker, social links of the speaker etc.
  9. sponsers json file consisting list of all sponsers of the event.
  10. tracks json file consisting of tracks specific data.
  11. config.json file which consists of the api url, app name.

After providing the required information we go to this site (http://droidgen.eventyay.com/) and the first thing this site asks us is the email id. Then we upload the required files mentioned above in a zip folder and we have a apk which we can test it out on our Android phone.

How did the Google IO sample app look like?

The files for the sample event can be found over here:

Folder Link:

https://github.com/fossasia/open-event/tree/master/sample/GoogleIO17

Zip File Link:

https://github.com/fossasia/open-event/blob/master/sample/GoogleIO17.zip

What were the issues found in the sample app?

There were certain issues which we observed on testing the app with the Google IO event:

  1. The theme of the app remains the same no matter which event it is. It is important to give the event organiser the ability to customise the theme of the app.
  2. The support for local speaker images needs to be provided as we want to give the event organiser an option to include the images locally or not.
  3. The background of the logo needs to be changed because in certain logos, the dark background causes visibility problems.
  4. Certain information in the app like the event information is hard-coded and needs to be taken from the assets folder instead of strings.xml.

Resources

Continue ReadingGenerating the Google IO Open Event Android App

Automatic Imports of Events to Open Event from online event sites with Query Server and Event Collect

One goal for the next version of the Open Event project is to allow an automatic import of events from various event listing sites. We will implement this using Open Event Import APIs and two additional modules: Query Server and Event Collect. The idea is to run the modules as micro-services or as stand-alone solutions.

Query Server
The query server is, as the name suggests, a query processor. As we are moving towards an API-centric approach for the server, query-server also has API endpoints (v1). Using this API you can get the data from the server in the mentioned format. The API itself is quite intuitive.

API to get data from query-server

GET /api/v1/search/<search-engine>/query=query&format=format

Sample Response Header

 Cache-Control: no-cache
 Connection: keep-alive
 Content-Length: 1395
 Content-Type: application/xml; charset=utf-8
 Date: Wed, 24 May 2017 08:33:42 GMT
 Server: Werkzeug/0.12.1 Python/2.7.13
 Via: 1.1 vegur

The server is built in Flask. The GitHub repository of the server contains a simple Bootstrap front-end, which is used as a testing ground for results. The query string calls the search engine result scraper scraper.py that is based on the scraper at searss. This scraper takes search engine, presently Google, Bing, DuckDuckGo and Yahoo as additional input and searches on that search engine. The output from the scraper, which can be in XML or in JSON depending on the API parameters is returned, while the search query is stored into MongoDB database with the query string indexing. This is done keeping in mind the capabilities to be added in order to use Kibana analyzing tools.

The frontend prettifies results with the help of PrismJS. The query-server will be used for initial listing of events from different search engines. This will be accessed through the following API.

The query server app can be accessed on heroku.

➢ api/list​: To provide with an initial list of events (titles and links) to be displayed on Open Event search results.

When an event is searched on Open Event, the query is passed on to query-server where a search is made by calling scraper.py with appending some details for better event hunting. Recent developments with Google include their event search feature. In the Google search app, event searches take over when Google detects that a user is looking for an event.

The feed from the scraper is parsed for events inside query server to generate a list containing Event Titles and Links. Each event in this list is then searched for in the database to check if it exists already. We will be using elastic search to achieve fuzzy searching for events in Open Event database as elastic search is planned for the API to be used.

One example of what we wish to achieve by implementing this type of search in the database follows. The user may search for

-Google Cloud Event Delhi
-Google Event, Delhi
-Google Cloud, Delhi
-google cloud delhi
-Google Cloud Onboard Delhi
-Google Delhi Cloud event

All these searches should match with “Google Cloud Onboard Event, Delhi” with good accuracy. After removing duplicates and events which already exist in the database from this list have been deleted, each event is rendered on search frontend of Open Event as a separate event. The user can click on any of these event, which will make a call to event collect.

Event Collect

The event collect project is developed as a separate module which has two parts

● Site specific scrapers
In its present state, event collect has scrapers for eventbrite and ticket-leap which, given a query, scrape eventbrite (and ticket-leap respectively) search results and downloads JSON files of each event using Loklak‘s API.
The scrapers can be developed in any form or any number of scrapers/scraping tools can be added as long as they are in alignment with the Open Event Import API’s data format. Writing tests for these against the concurrent API formats will take care of this. This part will be covered by using a json-validator​ to check against a pre-generated schema.

● REST APIs
The scrapers are exposed through a set of APIs, which will include, but not limited to,
➢ api/fetch-event : ​to scrape any event given the link and compose the data in a predefined JSON format which will be generated based on Open Event Import API. When this function is called on an event link, scrapers are invoked which collect event data such as event, meta, forms etc. This data will be validated against the generated JSON schema. The scraped JSON and directory structure for media files:
➢ api/export : to export all the JSON data containing event information into Open Event Server. As and when the scraping is complete, the data will be added into Open Event’s database as a new event.

How the Import works

The following graphic shows how the import works.




Let’s dive into the workflow. So as the diagram illustrates, the ‘search​’ functionality makes a call to api/list API endpoint provided by query-server which returns with events’ ‘Title’ and ‘Event Link’ from the parsed XML/JSON feed. This list is displayed as Open Event’s search results. Now the results having been displayed, the user can click on any of the events. When the user clicks on any event, the event is searched for in Open Event’s database. Two things happen now:

  • The event page loads if the event is found.
  • If the event does not already exist in the database, clicking on any event will

➢ Insert this event’s title and link in the database and get the event_id

➢ Make a call to api/fetch-event in event-collect which then invokes a site-specific scraper to fetch data about the event the user has chosen

➢ When the data is scraped, it is imported into Open Event database using the previously generated event_id. The page will be loaded using jquery ajax ​as and when the scraping is done.​When the imports are done, the search page refreshes with the new results. The Open Event Orga Server exposes a well documented REST API that can be used by external services to access the data.

Continue ReadingAutomatic Imports of Events to Open Event from online event sites with Query Server and Event Collect

Writing Simple Unit-Tests with JUnit

In the Loklak Server project, we use a number of automation tools like the build testing tool ‘TravisCI’, automated code reviewing tool ‘Codacy’, and ‘Gemnasium’. We are also using JUnit, a java-based unit-testing framework for writing automated Unit-Tests for java projects. It can be used to test methods to check their behaviour whenever there is any change in implementation. These unit-tests are handy and are coded specifically for the project. In the Loklak Server project it is used to test the web-scrapers. Generally JUnit is used to check if there is no change in behaviour of the methods, but in this project, it also helps in keeping check if the website code has been modified, affecting the data that is scraped.

Let’s start with basics, first by setting up, writing a simple Unit-Tests and then Test-Runners. Here we will refer how unit tests have been implemented in Loklak Server to familiarize with the JUnit Framework.

Setting-UP

Setting up JUnit with gradle is easy, You have to do just 2 things:-

1) Add JUnit dependency in build.gradle

Dependencies {

. . .

. . .<other compile groups>. . .

compile group: 'com.twitter', name: 'jsr166e', version: '1.1.0'

compile group: 'com.vividsolutions', name: 'jts', version: '1.13'

compile group: 'junit', name: 'junit', version: '4.12'

compile group: 'org.apache.logging.log4j', name: 'log4j-1.2-api', version: '2.6.2'

compile group: 'org.apache.logging.log4j', name: 'log4j-api', version: '2.6.2'

. . .

. . .

}

 

2) Add source for ‘test’ task from where tests are built (like here).

Save all tests in test directory and keep its internal directory structure identical to src directory structure. Now set the path in build.gradle so that they can be compiled.

sourceSets.test.java.srcDirs = ['test']

 

Writing Unit-Tests

In JUnit FrameWork a Unit-Test is a method that tests a particular behaviour of a section of code. Test methods are identified by annotation @Test.

Unit-Test implements methods of source files to test their behaviour. This can be done by fetching the output and comparing it with expected outputs.

The following test tests if twitter url that is created is valid or not that is to be scraped.

/**

 * This unit-test tests twitter url creation

 */

@Test

public void testPrepareSearchURL() {

String url;

String[] query = {"fossasia", "from:loklak_test",

"spacex since:2017-04-03 until:2017-04-05"};

String[] filter = {"video", "image", "video,image", "abc,video"};

String[] out_url = {

"https://twitter.com/search?f=tweets&vertical=default&q=fossasia&src=typd",

"https://twitter.com/search?f=tweets&vertical=default&q=from%3Aloklak_test&src=typd",

"and other output url strings to be matched…..."

};


// checking simple urls

for (int i = 0; i < query.length; i++) {

url = TwitterScraper.prepareSearchURL(query[i], "");


//compare urls with urls created

assertThat(out_url[i], is(url));

}


// checking urls having filters

for (int i = 0; i < filter.length; i++) {

url = TwitterScraper.prepareSearchURL(query[0], filter[i]);


//compare urls with urls created

assertThat(out_url[i+3], is(url));

}

}

 

Testing the implementation of code is useless as it will either make code more difficult to change or tests useless  . So be cautious while writing tests and keep difference between Implementation and Behaviour in mind.

This is the perfect example for a simple Unit-Test. As we see there are some points, which needs to be observed like:-

1) There is a annotation @Test .

2) Input array of query which is fed to the method TwitterScraper.prepareSearchURL() .

3) Array of urls out_url[], which are the expected urls to output.

4) asserThat() to compare the expected url (in array out_url[]) and the output url (in variable ‘url’).

NOTE: assertEquals() could also be used here, but we prefer to use assert methods to get error message that is readable (We will discuss about this some time later)

And the TestRunner

When we are working on a project, It is not feasible to run tests using gradle as they are first built  (else verified whether tests are build-ready) and then executed. gradle test shall be used only for building and testing the tests. For testing the project, one shall set-up TestRunner. It allows to run specific set of tests, one wants to run.

TestRunners are built once using gradle (with other tests) and can be run whenever you want. Also it is easy to stack up the test classes you want to run in SuiteClasses and @RunWith to run SuiteClasses with the TestRunner.

In loklak server, TestRunner runs the web-scraper tests. They are used by developers to test the changes they have made.

This is a sample TestRunner, code link here .

package org.loklak;


// Library classes imported

import org.junit.runner.RunWith;

import org.junit.runners.Suite;

// Source files to be tested

import org.loklak.harvester.TwitterScraperTest;

import org.loklak.harvester.YoutubeScraperTest;


/*

* TestRunner for harvesters

*/

@RunWith(Suite.class)

@Suite.SuiteClasses({

TwitterScraperTest.class,

YoutubeScraperTest.class

})

public class TestRunner {

}

 

You can also add TestRunners for different sections of the project. Like here it is initialized only to test harvesters.

To run the TestRunner

Add classpath of the jar file of the project and run ‘JUnitCore’ with TestRunner to get output on terminal.

java -classpath .:build/libs/<yourProject>.jar:build/classes/test org.junit.runner.JUnitCore org.loklak.TestRunner

In the project we have set up a shell script to run the tests.

Few points

1) Build the project and tests separately. Build tests only when changed as they take time to be built and executed.

2) Whenever you are done with the coding part, run the tests using TestRunner.

3) Write unit-tests whenever you add a new feature to the project to keep it up-to-date.

Now lets end up here.

So for now, Code it, Test it and Repeat.

Resources:

Continue ReadingWriting Simple Unit-Tests with JUnit