API

Read more about the article Implementing Loklak APIs in Java using Reflections

Implementing Loklak APIs in Java using Reflections

Loklak server provides a large API to play with the data scraped by it. Methods in java can be implemented to use these API endpoints. A common approach of implementing the methods for using API endpoints is to create the request URL by taking the values passed to the method, and then send GET/POST request. Creating the request URL in every method can be tiresome and in the long run maintaining the library if implemented this way will require a lot of effort. For example, assume a method is to be implemented for suggest API endpoint, which has many parameters, for creating request URL a lot of conditionals needs to be written - whether a parameter is provided or not. Well, the methods to call API endpoints can be implemented with lesser and easy to maintain code using Reflection in Java. The post ahead elaborates the problem, the approach to solve the problem and finally solution which is implemented in loklak_jlib_api. Let's say, the status API endpoint needs to be implemented, a simple approach can be: public class LoklakAPI { public static String status(String baseUrl) { String requestUrl = baseUrl "/api/status.json"; // GET request using requestUrl } public static void main(String[] argv) { JSONObject result = status("https://api.loklak.org"); } } This one is easy, isn’t it, as status API endpoint requires no parameters. But just imagine if a method implements an API endpoint that has a lot of parameters, and most of them are optional parameters. As a developer, you would like to provide methods that cover all the parameters of the API endpoint. For example, how a method would look like if it implements suggest API endpoint, the old SuggestClient implementation in loklak_jlib_api does that: public static ResultList<QueryEntry> suggest( final String hostServerUrl, final String query, final String source, final int count, final String order, final String orderBy, final int timezoneOffset, final String since, final String until, final String selectBy, final int random) throws JSONException, IOException { ResultList<QueryEntry> resultList = new ResultList<>(); String suggestApiUrl = hostServerUrl + SUGGEST_API + URLEncoder.encode(query.replace(' ', '+'), ENCODING) + PARAM_TIMEZONE_OFFSET + timezoneOffset + PARAM_COUNT + count + PARAM_SOURCE + (source == null ? PARAM_SOURCE_VALUE : source) + (order == null ? "" : (PARAM_ORDER + order)) + (orderBy == null ? "" : (PARAM_ORDER_BY + orderBy)) + (since == null ? "" : (PARAM_SINCE + since)) + (until == null ? "" : (PARAM_UNTIL + until)) + (selectBy == null ? "" : (PARAM_SELECT_BY + selectBy)) + (random < 0 ? "" : (PARAM_RANDOM + random)) + PARAM_MINIFIED + PARAM_MINIFIED_VALUE; // GET request using suggestApiUrl } } A lot of conditionals!!! The targeted users may also get irritated if they need to provide all the parameters every time even if they don’t need them. The obvious solution to that is overloading the methods. But,  then again for each overloaded method, the same repetitive conditionals need to be written, a form of code duplication!! And what if you have to implement some 30 API endpoints and in…

Continue ReadingImplementing Loklak APIs in Java using Reflections
Read more about the article Adding a new Servlet/API to SUSI Server for Skill Wiki

Adding a new Servlet/API to SUSI Server for Skill Wiki

Susi skill wiki is an editor to write and edit skill easily. It follows an API-centric approach where the Susi server acts as API server and a web front-end  act as the client for the API and provides the user interface. A skill is a set of intents. One text file represents one skill, it may contain several intents which all belong together. The schema for storing a skill is as following: Using this, one can access any skill based on four tuples parameters model, group, language, skill.  To achieve this on server side let’s create an API endpoint to list all skills based on given model, groups and languages. To check the source for this endpoint clone the susi_server repository from here. git clone https://github.com/fossasia/susi_server.git Have a look at documentation for more information about Susi Server. The Servlet java file is placed in susi_server/ai/susi/server/api/cms/ListSkillService. To implement the endpoint we will use the HttpServlet class which provides methods, such as doGet and doPost, for handling HTTP-specific services. In Susi Server an abstract class AbstractAPIHandler extending HttpServelets and implementing API handler interface is provided.  Next we will inherit our ListSkillService class from AbstractAPIHandler and implement APIhandler interface. To implement our servlet we will be overriding 4 methods namely Minimal Base User role  public BaseUserRole getMinimalBaseUserRole() { return BaseUserRole.ANONYMOUS; } This method tells the minimum Userrole required to access this servlet it can also be ADMIN, USER. In our case it is Anonymous. A User need not to log in to access this endpoint. Default Permissions   public JSONObject getDefaultPermissions(BaseUserRole baseUserRole) { return null; } This method returns the default permission attached with base user role, our servlets has nothing to do with it, therefore we can simply return null for this case. The API Path  public String getAPIPath() { return "/cms/getSkillList.json"; } This methods sets the API endpoint path, it gets appended to base path which is 127.0.0.1:4000/cms/getSkillList.json for the local host and http://api.susi.ai/cms/getSkillList.json for the server. The ServiceImpl method  public ServiceResponse serviceImpl(Query call, HttpServletResponse response, Authorization rights, final JsonObjectWithDefault permissions) { String model_name = call.get("model", "general"); File model = new File(DAO.model_watch_dir, model_name); String group_name = call.get("group", "knowledge"); File group = new File(model, group_name); String language_name = call.get("language", "en"); File language = new File(group, language_name); ArrayList fileList = new ArrayList(); fileList = listFilesForFolder(language, fileList); JSONArray jsArray = new JSONArray(fileList); JSONObject json = new JSONObject(true) .put("model", model_name) .put("group", group_name) .put("language", language_name) .put("skills", jsArray); return new ServiceResponse(json); } ArrayList listFilesForFolder(final File folder, ArrayList fileList) { File[] filesInFolder = folder.listFiles(); if (filesInFolder != null) { for (final File fileEntry : filesInFolder) { if (!fileEntry.isDirectory()) { fileList.add(fileEntry.getName()+""); } } } return fileList; } To access any skill we need parameters model, group, language. We get this through call.get method where first parameter is the key for which we want to get the value and second parameter is the default value. Based on received model, group and language browse files in that folder and put them in Json array to return the Service Json response. That’s all…

Continue ReadingAdding a new Servlet/API to SUSI Server for Skill Wiki

Implementing Voice Search In Susper (in Chrome only)

Last week @mariobehling opened up an issue to implement voice search in Susper. Google Chrome provides an API to integrate Speech recognition feature with any website. More about API can be read here: https://shapeshed.com/html5-speech-recognition-api/ The explanation might be in Javascript but it has been written following syntax of Angular 4 and Typescript. So, I created a speech-service including files: speech-service.ts speech-service.spec.ts Code for speech-service.ts: This is the code which will control the working of voice search. import { Injectable, NgZone } from '@angular/core'; import { Observable } from 'rxjs/Rx'; interface IWindow extends Window {   webkitSpeechRecognition: any; } @Injectable() export class SpeechService { constructor(private zone: NgZone) { } record(lang: string): Observable<string> {   return Observable.create(observe => {     const { webkitSpeechRecognition }: IWindow = <IWindow>window;     const recognition = new webkitSpeechRecognition();     recognition.continuous = true;     recognition.interimResults = true;     recognition.onresult = take => this.zone.run(() => observe.next(take.results.item(take.results.length - 1).item(0).transcript) );     recognition.onerror = err =>observe.error(err);     recognition.onend = () => observe.complete();     recognition.lang = lang;     recognition.start(); }); } } You can find more details about API following the link which I have provided above in starting. Here recognition.onend() => observe.complete() works as an important role here. Many developers forget to use it when working on voice search feature. It works like: whenever a user stops speaking, it will automatically understand that voice action has now been completed and the search can be attempted. And for this: speechRecognition() {   this.speech.record('en_US').subscribe(voice => this.onquery(voice)); } We have used speechRecognition() function. onquery() function is called when a query is entered in a search bar. Default language has been set up as 'en_US' i.e English. We also created an interface to link it with the API which Google Chrome provides for adding voice search feature on any website. I have also used a separate module by name NgZone. Now, what is NgZone? It is used as an injectable service for executing working inside or outside of the Angular zone. I won't go into detail about this module much here. More about it can be found on angular-docs website. We have also, implemented a microphone icon on search bar similar to Google. This is how Susper's homepage looks like now: This feature only works in Google Chrome browser and for Firefox it doesn't. So, for Firefox browser there was no need to show 'microphone' icon since voice search does not work Firefox. What we did simply use CSS code like this: @-moz-document url-prefix() {   .microphone {     display: none;   } } @-moz-document url-prefix() is used to target elements for Firefox browser only. Hence using, this feature we made it possible to hide microphone icon from Firefox and make it appear in Chrome. For first time users: To use voice search feature click on the microphone feature which will trigger speechRecognition() function and will ask you permission to allow your laptop/desktop microphone to detect your voice. Once allowing it, we're done! Now the user can easily, use voice search feature on Susper to search for a random thing.

Continue ReadingImplementing Voice Search In Susper (in Chrome only)

How to add a new Servlet/API to SUSI Server

You have got a new feature added to enhance SUSI-AI (in web/android/iOS application) but do not find an API which could assist you in your work to make calls to the server {since the principle of all Susi-AI clients is to contact with SUSI-server for any feature}. Making servlets for  Susi is quite different from a normal JAVA servlet. Though the real working logic remains the same but we have got classes which allow you to directly focus on one thing and that is to maintain your flow for the feature. To find already implemented servlets, first clone the susi_server repository  from here. git clone https://github.com/fossasia/susi_server.git Cd to susi_server directory or open your terminal in susi_server directory. (This blog focuses on servlet development for Susi only and hence it is assumed that you have any version of JAVA8 installed properly). If you have not gone through how to run a susi_server manually, then follow  below steps to start the server: ./gradlew build //some set of files and dependencies will be downloaded bin/start.sh //command to start the server This will start your Susi server and it will listen at port 4000. The first step is to analyze that to which class of API is your  servlet  going to be added. Let us take a small example and see how to proceed step by step. Let us look at development of ListSettingsService servlet. (to find the code of this servlet, browse to the following location: susi_server->src->ai->susi->server->api->aaa). Once you have decided the classification of your srvlet, create a .java file in it (Like we created ListSettingsService.java file in aaa folder). Extend AbstractAPIHandler class to your class and implement APIHandler to your class. If you are using any IDE like Intelij IDEA or eclipse then they will give you an error message and when you click on it, it  will ask you to Override some methods. Select the option and if you are using a simple text editor, then override the following methods in the given way: @Override public String getAPIPath() { return null; } @Override public BaseUserRole getMinimalBaseUserRole() { return null; } @Override public JSONObject getDefaultPermissions(BaseUserRole baseUserRole) { return null; } @Override public ServiceResponse serviceImpl(Query post, HttpServletResponse response, Authorization rights, JsonObjectWithDefault permissions) throws APIException { Return null; } What all these methods are for and why do we need them? These are those 4 methods that make our work way easy. With the code compilation, first getAPIPath() is called to evaluate the end point.  Whenever this end point is called properly, it responds with whatever is defined in serviceImpl(). In our case we have given the endpoint "/aaa/listSettings.json". Ensure that you do not have 2  servlets with same end point. Next in the line is getMinimalBaseUserRole() method. While developing certain features, a need of special privilege {like admin login} might be required. If you are implementing a feature for Admins only (like  we are doing in this servlet), return BaseUserRole.ADMIN. If you want to give access to anyone (registered or not) then return…

Continue ReadingHow to add a new Servlet/API to SUSI Server

Integrating an Image Editing Page in Phimpme Android

The main aim of the Phimpme is to develop image editing and sharing application as an alternative to proprietary solutions like Instagram. Any user can choose a photo from the gallery or click a picture from the camera and upload it on the various social media platform including Drupal and wordpress. As most of the image editor applications in the app store currently my team and I discussed and listed down the basic functionality of the Image editing activity. We have listed down the following features for image Editing activity: Filters. Stickers Image tuning Choosing the Image Editing Application There are number of existing Open Source projects that we went through to check how they could be integrated into Phimpme. We looked into those projects which are licensed under the  MIT Licence. As per the MIT Licence the user has the liberty to modify the use the code, modify it, merge, publish it without any restrictions. Image-Editor Android is one such application which has MIT Licence. The Image-Editor Android has extensive features for manipulating and enhancing the image. The features are as follows: Edit Image by drawing on it. Applying stickers on the image. Applying filters. Crop. Rotating the image. Text on the image. It is an ideal application to be implemented in our project. The basic flow of the application First, getting the image either by gallery or camera. The team has implemented leafPic and openCamera. Second, redirecting the image from the leafPic gallery to the Image editing activity by choosing edit option from the popup menu. Populating the Menu in the popup menu in XML: <menu> tag is the root node, which contains ites in the popup menu. The following code is used to populate the menu: <?xml version="1.0" encoding="utf-8"?> <menu xmlns:android="http://schemas.android.com/apk/res/android">    <item android:id="@+id/action_edit"          android:icon="@drawable/ic_edit"          android:title="@string/Edit"          android:showAsAction="ifRoom"/>    <item android:id="@+id/action_use_as"          android:icon="@drawable/ic_use_as"          android:title="@string/useAs" /> </menu> Setting up the Image Editing Activity Image-Editor Android application contains two main sections. MainActivity (To get the image). imageeditlibrary(To edit the image) We need to import imageeditlibrary module. Android studios gives easy method to import a module from any other project using GUI. This can be done as follows: File->new->import module then choosing the module from the desired application. Things to remember after importing any module from any other project: Making sure that the minSdkVersion and targetSdkVersion in the gradle of the imported module and the current working project is same. In Phimpme the minSdkVersion is 16 and tagetSdkVersion is 25, which is used as standard SDK version. Importing all the classes in the used in the imageeditlibrary module before using them in the leadPic gallery. Sending Image to Image Editing Activity This includes three tasks: Handling onclick listeners. Sending the image from the leafPic Activity Receiving the the image in EditImageActivity. Handling onClick Listener: public boolean onOptionsItemSelected(MenuItem item) {        switch (item.getItemId()) { case R.id.action_edit: // setOnclick listener here.  } } Sending Image to EditImageActivity: First we need to get the path of the image to be send. For this we need FileUtils class to…

Continue ReadingIntegrating an Image Editing Page in Phimpme Android

Implementing a chatbot using the SUSI.AI API

SUSI AI is an intelligent Open Source personal assistant. It is a server application which is able to interact with humans as a personal assistant. The first step in implementing a bot using SUSI AI is to specify the pathway for query response from SUSI AI server. The steps mentioned below provide a step-by-step guide to establish communication with SUSI AI server: Given below is HTML code that demonstrates how to connect with SUSI API through an AJAX call. To put this file on a Node Js server, see Step 2.  To view the response of this call, follow Step 4. <!DOCTYPE html> <body> <h1>My Header</h1> <p>My paragraph.</p> //Script with source here //Script to be written here </body> </html> In above code add scripts given below and end each script with closing tag </script>. In the second script we are calling SUSI API with hello query and showing data that we are receiving through call on console. <script src="https://ajax.googleapis.com/ajax/libs/jquery/3.2.1/jquery.min.js"> <script> $(function (){ $.ajax({ dataType: 'jsonp', type:'GET', url: 'http://api.susi.ai/susi/chat.json? timezoneOffset=-300&q=hello', success: function(data){ console.log('success', data); } }); }); Code below is in node js to setup localhost and getting the same above result on browser. Below is Node Js code to setup a server at localhost for the above created HTML file. var http = require('http'); var fs = require('fs'); http.createServer(function (req, res) {  fs.readFile('YOURFILENAME.html', function(err, data) {    res.writeHead(200, {'Content-Type': 'text/html'});    res.write(data);    res.end();  }); }).listen(9000); We will get following response by running this Node js code and checking results on http://localhost:9000/ To run this code install Node Js and write “node filename.js” in command line. You can open above window by right clicking on page and selecting Inspect. Go to the Network option and select the relevant api call from left section of Inspect window. We have successfully got response from SUSI API and now we can use this response for building bots for receiving replies for user.

Continue ReadingImplementing a chatbot using the SUSI.AI API
Read more about the article Generating the Google IO Open Event Android App

Generating the Google IO Open Event Android App

The main aim of FOSSASIA Open Event Android App is to give an event organiser the ability to generate the app through a single click by providing the necessary json and binary files. As of late the Android application was tested on Google IO 2017 event. The sample files can be seen here. The data with respect to the event was taken from this site (https://events.google.com/io/). What was astonishing about this application is the simplicity with which we can make an event specific application by giving the vital assets required (json and binary files). What was needed for generating the Google IO 2017 app? For generating the app we had to provide the following files: images folder containing the necessary images of speaker, the logo of the event etc. event json file which has all the event specific information like the name of the event, the schedule of the event, the description of the event etc. forms json file having session and speaker form data. meta json file having the root url of the event. microlocations json file having all the locations where the events are going to happen. session_types json file consisting data of all the type of session which will occur in the vent. sessions json file consisting session specific data like the title of the session, start time and end time of session, which track that session belongs to etc. speakers json file consisting of speaker specific data like the name of the speaker, image of the speaker, social links of the speaker etc. sponsers json file consisting list of all sponsers of the event. tracks json file consisting of tracks specific data. config.json file which consists of the api url, app name. After providing the required information we go to this site (http://droidgen.eventyay.com/) and the first thing this site asks us is the email id. Then we upload the required files mentioned above in a zip folder and we have a apk which we can test it out on our Android phone. How did the Google IO sample app look like? The files for the sample event can be found over here: Folder Link: https://github.com/fossasia/open-event/tree/master/sample/GoogleIO17 Zip File Link: https://github.com/fossasia/open-event/blob/master/sample/GoogleIO17.zip What were the issues found in the sample app? There were certain issues which we observed on testing the app with the Google IO event: The theme of the app remains the same no matter which event it is. It is important to give the event organiser the ability to customise the theme of the app. The support for local speaker images needs to be provided as we want to give the event organiser an option to include the images locally or not. The background of the logo needs to be changed because in certain logos, the dark background causes visibility problems. Certain information in the app like the event information is hard-coded and needs to be taken from the assets folder instead of strings.xml. Resources This tool makes work a lot easier by generating JSON files…

Continue ReadingGenerating the Google IO Open Event Android App

Automatic Imports of Events to Open Event from online event sites with Query Server and Event Collect

One goal for the next version of the Open Event project is to allow an automatic import of events from various event listing sites. We will implement this using Open Event Import APIs and two additional modules: Query Server and Event Collect. The idea is to run the modules as micro-services or as stand-alone solutions. Query Server The query server is, as the name suggests, a query processor. As we are moving towards an API-centric approach for the server, query-server also has API endpoints (v1). Using this API you can get the data from the server in the mentioned format. The API itself is quite intuitive. API to get data from query-server GET /api/v1/search/<search-engine>/query=query&format=format Sample Response Header Cache-Control: no-cache Connection: keep-alive Content-Length: 1395 Content-Type: application/xml; charset=utf-8 Date: Wed, 24 May 2017 08:33:42 GMT Server: Werkzeug/0.12.1 Python/2.7.13 Via: 1.1 vegur The server is built in Flask. The GitHub repository of the server contains a simple Bootstrap front-end, which is used as a testing ground for results. The query string calls the search engine result scraper scraper.py that is based on the scraper at searss. This scraper takes search engine, presently Google, Bing, DuckDuckGo and Yahoo as additional input and searches on that search engine. The output from the scraper, which can be in XML or in JSON depending on the API parameters is returned, while the search query is stored into MongoDB database with the query string indexing. This is done keeping in mind the capabilities to be added in order to use Kibana analyzing tools. The frontend prettifies results with the help of PrismJS. The query-server will be used for initial listing of events from different search engines. This will be accessed through the following API. The query server app can be accessed on heroku. ➢ api/list​: To provide with an initial list of events (titles and links) to be displayed on Open Event search results. When an event is searched on Open Event, the query is passed on to query-server where a search is made by calling scraper.py with appending some details for better event hunting. Recent developments with Google include their event search feature. In the Google search app, event searches take over when Google detects that a user is looking for an event. The feed from the scraper is parsed for events inside query server to generate a list containing Event Titles and Links. Each event in this list is then searched for in the database to check if it exists already. We will be using elastic search to achieve fuzzy searching for events in Open Event database as elastic search is planned for the API to be used. One example of what we wish to achieve by implementing this type of search in the database follows. The user may search for -Google Cloud Event Delhi -Google Event, Delhi -Google Cloud, Delhi -google cloud delhi -Google Cloud Onboard Delhi -Google Delhi Cloud event All these searches should match with “Google Cloud Onboard Event, Delhi” with good accuracy.…

Continue ReadingAutomatic Imports of Events to Open Event from online event sites with Query Server and Event Collect
Read more about the article Writing Simple Unit-Tests with JUnit

Writing Simple Unit-Tests with JUnit

In the Loklak Server project, we use a number of automation tools like the build testing tool ‘TravisCI’, automated code reviewing tool ‘Codacy’, and ‘Gemnasium’. We are also using JUnit, a java-based unit-testing framework for writing automated Unit-Tests for java projects. It can be used to test methods to check their behaviour whenever there is any change in implementation. These unit-tests are handy and are coded specifically for the project. In the Loklak Server project it is used to test the web-scrapers. Generally JUnit is used to check if there is no change in behaviour of the methods, but in this project, it also helps in keeping check if the website code has been modified, affecting the data that is scraped. Let’s start with basics, first by setting up, writing a simple Unit-Tests and then Test-Runners. Here we will refer how unit tests have been implemented in Loklak Server to familiarize with the JUnit Framework. Setting-UP Setting up JUnit with gradle is easy, You have to do just 2 things:- 1) Add JUnit dependency in build.gradle Dependencies { . . . . . .<other compile groups>. . . compile group: 'com.twitter', name: 'jsr166e', version: '1.1.0' compile group: 'com.vividsolutions', name: 'jts', version: '1.13' compile group: 'junit', name: 'junit', version: '4.12' compile group: 'org.apache.logging.log4j', name: 'log4j-1.2-api', version: '2.6.2' compile group: 'org.apache.logging.log4j', name: 'log4j-api', version: '2.6.2' . . . . . . }   2) Add source for 'test' task from where tests are built (like here). Save all tests in test directory and keep its internal directory structure identical to src directory structure. Now set the path in build.gradle so that they can be compiled. sourceSets.test.java.srcDirs = ['test']   Writing Unit-Tests In JUnit FrameWork a Unit-Test is a method that tests a particular behaviour of a section of code. Test methods are identified by annotation @Test. Unit-Test implements methods of source files to test their behaviour. This can be done by fetching the output and comparing it with expected outputs. The following test tests if twitter url that is created is valid or not that is to be scraped. /** * This unit-test tests twitter url creation */ @Test public void testPrepareSearchURL() { String url; String[] query = {"fossasia", "from:loklak_test", "spacex since:2017-04-03 until:2017-04-05"}; String[] filter = {"video", "image", "video,image", "abc,video"}; String[] out_url = { "https://twitter.com/search?f=tweets&vertical=default&q=fossasia&src=typd", "https://twitter.com/search?f=tweets&vertical=default&q=from%3Aloklak_test&src=typd", "and other output url strings to be matched…..." }; // checking simple urls for (int i = 0; i < query.length; i++) { url = TwitterScraper.prepareSearchURL(query[i], ""); //compare urls with urls created assertThat(out_url[i], is(url)); } // checking urls having filters for (int i = 0; i < filter.length; i++) { url = TwitterScraper.prepareSearchURL(query[0], filter[i]); //compare urls with urls created assertThat(out_url[i+3], is(url)); } }   Testing the implementation of code is useless as it will either make code more difficult to change or tests useless  . So be cautious while writing tests and keep difference between Implementation and Behaviour in mind. This is the perfect example for a simple Unit-Test. As we see there are some points,…

Continue ReadingWriting Simple Unit-Tests with JUnit