GSoC

Read more about the article Save Server Response to File Using Python in Open Event Android App Generator

Save Server Response to File Using Python in Open Event Android App Generator

The Open Event Android project helps event organizers to generate Apps (apk format) using Open Event App generator for their events/conferences by providing API endpoint or zip generated using Open Event server.

The Open Event Android project has an assets folder which contains sample JSON files for the event, which are used to load the data when there is no Internet connection. When an organizer generates an app with the help of an API endpoint, then all the assets, e.g. images, are removed and because of it generated don’t have sample JSON files. So if the device is offline then it will not load the data from the assets (issue #1606). The app should contain sample event JSON files and should be able to load the data without the Internet connection.

One solution to this problem is to fetch all the event data (JSON files) from the server and save it to the assets folder while generating the app using Open Event App generator. So in this blog post, I explain how to make a simple request using Python and save the response in a file.

Making a simple request

To fetch the data we need to make a request to the server and the server will return the response for that request. Here’s step to make a simple request,

Import Requests module using,

import requests

The requests module has get, post, put, delete, head, options to make a request. Example:

requests.get(self.api_link + '/' + end_point)

Here all method returns requests.Response object. We can get all the information we need from this object.

So we can make response object like this,

response = requests.get(self.api_link + '/' + end_point)

Saving response in a file

For saving response in a file we need to open/create a file for writing.

Create or open file

The open() method is used for opening file. It takes two arguments one is file name and second in access mode which determines the mode in which the file has to be opened, i.e., read, write, append, etc.

file = open(path/to/the/file, "w+")

Here “w+” overwrites the existing file if the file exists. If the file does not exist, it creates a new file for reading and writing.

Write and close file

Now write the content of the response using the write() method of file object then close the file object using the close()  method. The close() method flushes any unwritten information and closes the file object.

file.write(response.text)
file.close()

Here response.text gives the content of the response in Unicode format.

Conclusion

The requests.Response object contains much more data like encoding, status_code, headers etc. For more info about request and response refer additional resources given below.

Additional Resources

Continue ReadingSave Server Response to File Using Python in Open Event Android App Generator
Read more about the article JSON Deserialization Using Jackson in Open Event Android App

JSON Deserialization Using Jackson in Open Event Android App

The Open Event project uses JSON format for transferring event information like tracks, sessions, microlocations and other. The event exported in the zip format from the Open Event server also contains the data in JSON format. The Open Event Android application uses this JSON data. Before we use this data in the app, we have to parse the data to get Java objects that can be used for populating views. Deserialization is the process of converting JSON data to Java objects. In this post I explain how to deserialize JSON data using Jackson.

1. Add dependency

In order to use Jackson in your app add following dependencies in your app module’s build.gradle file.

dependencies {
	compile 'com.fasterxml.jackson.core:jackson-core:2.8.9'
	compile 'com.fasterxml.jackson.core:jackson-annotations:2.8.9'
	compile 'com.fasterxml.jackson.core:jackson-databind:2.8.9'
}

2.  Define entity model of data

In the Open Event Android we have so many models like event, session, track, microlocation, speaker etc. Here i am only defining track model because of it’s simplicity and less complexity.

public class Track {

	private int id;
	private String name;
	private String description;
	private String color;
	@JsonProperty("font-color")       
	private String fontColor;
    
	//getters and setters
}

Here if the property name is same as json attribute key then no need to add JsonProperty annotation like we have done for id, name color property. But if property name is different from json attribute key then it is necessary to add JsonProperty annotation.

3.  Create sample JSON data

Let’s create sample JSON format data we want to deserialize.

{
        "id": 273,
        "name": "Android",
        "description": "Sample track",
        "color": "#94868c",
        "font-color": "#000000"
}

4.  Deserialize using ObjectMapper

ObjectMapper is Jackson serializer/deserializer. ObjectMapper’s readValue() method is used for simple deserialization. It takes two parameters one is JSON data we want to deserialize and second is Model entity class. Create an ObjectMapper object and initialize it.

ObjectMapper objectMapper = new ObjectMapper();

Now create a Model entity object and initialize it with deserialized data from ObjectMapper’s readValue() method.

Track track = objectMapper.readValue(json, Track.class);

So we have converted JSON data into the Java object.

Jackson is very powerful library for JSON serialization and deserialization. To learn more about Jackson features follow the links given below.

Continue ReadingJSON Deserialization Using Jackson in Open Event Android App
Read more about the article Debugging JSON Files of Sample Events for Open Event Android using Stetho

Debugging JSON Files of Sample Events for Open Event Android using Stetho

In this blog, I will talk about data validation and debugging in Android using the Stetho-A debug bridge for Android by Facebook. Most Android applications have JSON files to populate the elements like RecyclerViews, ListViews and different types of Layouts. Stetho is a debug tool for Android which uses the well-known Chrome Developer tools as it’s User Interface. With Stetho, we can see all our incoming JSON data in spreadsheets making debugging much easier and fun. What’s more, it’s completely Open source.

Getting started with Stetho

To start with, we need to enable Stetho by adding it as one of the dependencies in the build.gradle file.

compile 'com.facebook.stetho:stetho:1.4.2'

It is already enabled in the Open Event Android app.

Now, you need to add the api endpoint for your sample in the config.json file in the project(if you are using the Android repo for debugging). The config.json file is at app/assets/

In case, you are using an Open Event generated app then there’s no need to do that.

Now connect your phone through the USB cable to your laptop and start your application.

Next, you have to browse to chrome://inspect where you will see your device in the window as shown below.

 

On clicking “Inspect”, you will be shown the Stetho debug tool interface. Something like this:

When you download the details by clicking “Yes” on the starting of the application, be sure to keep it connected to the Stetho tool in the ‘inspect’ mode.

Once the data has loaded, go to Web SQL-> default.realm. You will see tabs like:-

class_speaker, class_sponsor, class_session, class_track among others.

On clicking them, you will see well-organised tables that show all linked attributes of the class that you selected.

Sessions can be seen along with all the related information in a tabular format.

Stetho displays speakers of the event and their information in an easy-to-read tabular format.

Micro locations of the event along with related information in a tabular format.

This is how you can view all the attributes of your sample in a tabular layout and hence, debug them easily. Although, Stetho is about much more than this, this is all I will talk about in this blog.

You can read more about Stetho and it’s functionalities here.

The official repo for Stetho can be found here for the source code.

Continue ReadingDebugging JSON Files of Sample Events for Open Event Android using Stetho
Read more about the article Cache Thumbnails and Images Using Picasso in Open Event Android

Cache Thumbnails and Images Using Picasso in Open Event Android

In the event based Android projects like Open Event Android, we have speakers and sponsors. And these projects needs to display image of the speakers and sponsors because it affects project a lot. So instead of every time fetching image from the server it is good to store small images(thumbnails) in the cache and load images even if device is offline. It also reduces data usage.

Picasso is mostly used image loading library for Android. It automatically handles ImageView recycling and download cancellation in an adapter, complex image transformations with minimal memory use, memory and disk caching.

But one problem is Picasso caches images for only one session by default. I mean if you close the app then all by default cached image will be removed.  If you are offline then Picasso will not load cached images because of it. It will make network calls every time you open the app.

In this post I explain how to manually cache images using Picasso so that images load even if the device is offline. It will make a network call only once for a particular image and will cache image in memory.

We will use okhttp3 library for OkHttpClient.

1. Add dependency

In order to use Picasso in your app add following dependencies in your app module’s build.gradle file.

dependencies {
        compile 'com.squareup.okhttp3:okhttp:3.8.1'
        compile 'com.squareup.picasso:picasso:2.5.2'
}

2. Make static Picasso object

Make static Picasso object in the Application class so that we can use it directly from the other activity.

public static Picasso picassoWithCache;

3. Initialize cache

Create a File object with path as app specific cache and use this object to create a Cache object.

File httpCacheDirectory = new File(getCacheDir(), "picasso-cache");
Cache cache = new Cache(httpCacheDirectory, 15 * 1024 * 1024);

Here it will create a Cache object with 15MB. getCacheDir() method returns the absolute path to the application specific cache directory on the filesystem.

OkHttpClient.Builder okHttpClientBuilder = new OkHttpClient.Builder().cache(cache);

4. Initialize Picasso with cache

Now initialize picassoWithCache object using Picass.Builder(). Set downloader for picasso by adding  new OkHttp3Downloader object.

picassoWithCache = new Picasso.Builder(this).downloader(new OkHttp3Downloader(okHttpClientBuilder.build())).build();

5. Use picassoWithCache object

As it is a static object you can directly use it from any activity. All the images loaded using this picassoWithCache instance will be cached in memory.

Application.picassoWithCache().load(thumbnail/image url);

 

To know more how i solved this issue in Open Event Project visit this link. To learn more about Picasso features follow the links given below.

Continue ReadingCache Thumbnails and Images Using Picasso in Open Event Android
Read more about the article Displaying selective logs in Yaydoc with downloadable detailed logs

Displaying selective logs in Yaydoc with downloadable detailed logs

Yaydoc, our automatic documentation generation and deployment project, at the crux of it consists of bash scripts. These bash scripts perform various actions, including documentation generation, deployment to Github and deployment to Heroku.

Since the inception of the User Interface of the Web UI, we have been spitting out the output of the bash script directly to the console output block without any filter. We understand that the output contains a lot of jargon that provides no essential information to the user. Hence, to improve the user experience of our platform, we decided to separate the Detailed logs and show only basic logs outlining the flow of the processes.

To implement this, we append all the logs to a unique file, sending only basic logs to stdout and stderr. Since we attend to store logs and display them in the console block of Web UI simultaneously we use the tee command, piping it with echo commands.

function print_log {
  if [ -n “$LOGFILE” ]; then
    echo -e $1 | tee -a ${LOGFILE}
  else
    echo -e $1
  fi
}

${LOGFILE}  is a unique file that has the same name as the preview directory and the compressed repository. After storing the logs in the file, the contents of the file are outputted using the cat command and is then shown to the user in a modal which is after the Detailed Logs button is clicked.

$(“#btnLogs”).click(function () {
  socket.emit(‘retrieve-detailed-logs’, data);
  ....
});

socket.on(‘retrieve-detailed-logs’, function (data) {
  var process = spawn(‘cat’, [‘temp/’+data.email+’/’+data.uniqueId+’.txt’]);
  process.stdout.setEncoding(‘utf-8’);
  process.stdout.on(‘data’, function (data)) {
    socket.emit(‘file-content’, data);
  }
});

To keep the indentation of the logs, we display the content inside the HTML pre tags. Along with displaying the detailed logs, we also let our two additional functionalities. These are ‘Copy to Clipboard’ and ‘Download as text file’. The ‘copy to clipboard’ functionality is achieved using the clipboard.js jQuery library while the `Download` functionality is achieved using res.download(file) function of ExpressJS response.

socket.on(‘file-content’, function (data) {
  new Clipboard(‘#copy-button’);
  $(‘#detailed-logs’).html(data);
  $(‘#detailed-logs-modal’).modal(‘show’);
});

Resources:

  1. https://clipboardjs.com – A modern approach to copy text to clipboard
  2. https://socket.io/ – The fastest and most reliable real-time engine
Continue ReadingDisplaying selective logs in Yaydoc with downloadable detailed logs
Read more about the article Creating A Sample Event Web App Using the Open Event Framework

Creating A Sample Event Web App Using the Open Event Framework

I have been creating sample events for the Open Event Framework by creating JSON files for them. In this blog, I will walk you through 4 easy steps to create a sample event for FOSSASIA Open Event. These are the steps that I followed while I contributed in making sample web apps:

Let’s start by taking the example of a sample event. I recently created a sample for FBF8’2017. F8 is conference held by Facebook annually.

I started by visiting the official F8 site. I personally used Google Chrome for it as I was almost positive that the official site will be having an API which will make it easier for me to extract JSON for the event. Also, Google Chrome developer tools make it easier to access the API. I have previously explained how to access JSON through APIs in one of my previous blog posts that can be found here.

So in the F8 site, I looked for the page that will lead me to the full schedule of the conference:

After clicking on the “SEE FULL SCHEDULE” button, this is where it leads us, with the developer tools ON.

That weird named file on that appears in under the XHR tab contains our JSONs!

[{1492502400: [{day: 1, sessionTitle: "Registration", sessionSlug: "day-1-registration",…},…],…}, {,…}]

0:{1492502400: [{day: 1, sessionTitle: "Registration", sessionSlug: "day-1-registration",…},…],…}

1:{,…}

This is how the file looks on clicking it at first. You must be seeing small clickable arrows on the sides which expand the JSON to reveal all the contents. But okay, we have crossed our first step: Obtaining JSON from the website’s API!

Step 2: Tailoring the JSON according to the Open Event format needs to be done now. To view the latest API format, visit here. Now this step is obviously most time-consuming and tedious. We need to modify the API we obtained from the F8 site to meet the Open Event requirements.

I mainly used Sublime text editor to do this. It has this great feature to find and replace text in text files.

Taking an example of a session object. This is what we will get from the API:-

allDay:false

createdAt:"2017–01–28T01:24:12.770Z"

day:1

displayTime:"3:00pm"

endTime:{__type: "Date", iso: "2017–04–18T15:50:00.000Z"}

iso:"2017–04–18T15:50:00.000Z"

__type:"Date"

hasDetails:true

objectId:"62yCeJMvAf"

ogImage:{__type: "File", name: "296643cca9aad196bbd3534324c52049_Making Great Camera Effects.png",…}

name:"296643cca9aad196bbd3534324c52049_Making Great Camera Effects.png"

url:"https://f82017-parse.s3.amazonaws.com/296643cca9aad196bbd3534324c52049_Making%20Great%20Camera%20Effects.png"

__type:"File"

onMySchedule:false

sessionDescription:"With AI that transforms images, Facebook's Camera Effects Platform lets you create animated masks, interactive overlays, frames, and other effects that people can apply to their photos and videos in the Facebook camera. Learn best technical and design practices for creating effects using this innovative technology, and see examples of successful effects."

sessionLocation:"G"

sessionSlug:"making-great-camera-effects"

sessionTitle:"Making Great Camera Effects"

sortTime:"1492527600"

speakers:[{speakerName: "Volodymyr Giginiak", createdAt: "2017–01–28T01:27:49.292Z",…},…]

0:{speakerName: "Volodymyr Giginiak", createdAt: "2017–01–28T01:27:49.292Z",…}

1:{speakerName: "Stef Smet", createdAt: "2017–01–28T01:30:16.022Z",…}

startTime:{__type: "Date", iso: "2017–04–18T15:00:00.000Z"}

iso:"2017–04–18T15:00:00.000Z"

__type:"Date"

tags:["Camera"]

0:"Camera"

updatedAt:"2017–04–18T19:19:16.501Z"

Now, this is where I used the find & replace feature in Sublime text to make it look like the open-event standard JSON. For instance, replacing “sessionLocation” with “microlocation”, “sessionDescription” with “long_abstract” and so on. Yes, there were a few manual changes as well that I had to make.

This is how the final JSON session object looked like after completing step 2:

{
     "short_abstract": "",
     "id": 26,
     "subtitle": "making-great-camera-effects",
     "title": "Making Great Camera Effects",
     "long_abstract": "With AI that transforms images, Facebook's Camera Effects Platform lets you create animated masks, interactive overlays, frames, and other effects that people can apply to their photos and videos in the Facebook camera. Learn best technical and design practices for creating effects using this innovative technology, and see examples of successful effects.",
     "end_time": "2017-04-18T15:50:00-07:00",
     "microlocation": {
        "id": 5,
        "name": "Hall G"
     },
     "start_time": "2017-04-18T15:00:00-07:00",
     "comments": "",
     "language": "",
     "slides": "",
     "audio": "",
     "video": "",
     "signup_url": "",
     "state": "accepted",
     "session_type": {
        "id": 6,
        "name": "50 minutes session",
        "length": "00.50"
     },
     "track": {
        "id": 2,
        "name": "Camera",
        "color": "#FF3046"
     },
     "speakers": [{
           "city": "",
           "heard_from": "",
           "icon": "",
           "long_biography": "",
           "small": "",
           "photo": "",
           "thumbnail": "",
           "short_biography": "",
           "speaking_experience": "",
           "sponsorship_required": "",
           "organisation": "Facebook",
           "id": 40,
           "name": "Volodymyr Giginiak"
        }
     ]
  }

Apart from this, there were places where I had to dig for information about speakers, their social website links, their images and so on because the open-event JSON format expects all this.

Step 3: JSON Validation

It is very likely that one of your JSON files will contain invalid JSON strings. So it is important to validate them. I used this online tool and I liked it very much. Once the JSON is validated, everything is set. All you need to now do is-move to step 4!

Step 4: Web app generation

Compress your JSON archive in .zip format and then go to the webapp generator. Enter your email ID and upload the zip file. Now, you can deploy, download and preview your personalised web-app, generated by FOSSASIA open-event web app generator.

The official web-app generator repo is here.

If you want to contribute to FOSSASIA by creating a sample, you can create an issue in the official open-event repo, along with a PR which contains the JSON sample files, the zipped file and a link to the web-app deployment on GitHub pages.

Continue ReadingCreating A Sample Event Web App Using the Open Event Framework
Read more about the article Create Scraper in Javascript for Loklak Scraper JS

Create Scraper in Javascript for Loklak Scraper JS

Loklak Scraper JS is the latest repository in Loklak project. It is one of the interesting projects because of expected benefits of Javascript in web scraping. It has a Node Javascript engine and is used in Loklak Wok project as bundled package. It has potential to be used in different repositories and enhance them.

Scraping in Python is easy (at least for Pythonistas) as one needs to just import Request library and BeautifulSoup library (lxml as better option), write some lines of code using Request library to get webpage and some lines of bs4 to walk through html and scrape data. This sums up to about less than a hundred lines of coding, where as Javascript coding isn’t easily readable (at least to me) as compared to Python. But it has an advantage, it can easily deal with Javascript in the pages we are scraping. This is one of the motive, Loklak Scraper JS repository was created and we contributed and worked on it.

I recently coded a Javascript scraper in loklak_scraper_js repository. While coding, I found it’s libraries similar to the libraries, I use to code in Python. Therefore, this blog is for Pythonistas how they can start scraping in Javascript as they finish reading and also contribute to Loklak Scraper JS.

First, replace Python interpreter, Request and Beautifulsoup library with Node JS interpreter, Request and Cheerio JS library.

1) Node JS Interpreter: Node JS Interpreter is used to interpret Javascript files. This is different from Python as it deals with the project instead of a module in case of Python. The most compatible Node for most of the libraries is 6.0.0 , where as latest version available(as I checked) is 8.0.0

TIP: use `–save` with npm like here while installing a library.

2) Request Library :- This is used to load webpage to be processed. Similar to one in Python.

Request-promise library, a wrapper around Request with implementation of Bluebird library, improves readability and makes code cleaner (how?).

 

3) Cheerio Library:- A Pythonista (a rookie one) can call it twin of BeautifulSoup Library. But this is faster and is Javascript. It’s selector implementation is nearly identical to jQuery’s.

Let us code a basic Javascript scraper. I will take TimeAndDate scraper from loklak_scraper_js as example here. It inputs place and outputs its local time.

Step#1: fetching HTML from webpage with the help of Request library.

We input url to Request function to fetch the webpage and is saved to `html` variable. This scrapeTimeAndDate() function scrapes data from html

url = "http://www.timeanddate.com/worldclock/results.html?query=London";

request(url, function(error, response, body) {

 if(error) {

    console.log("Error: " + error);

    process.exit(-1);

 }

 html = body;

 scrapeTimeAndDate()

});

 

Step#2: To scrape important data from html using Cheerio JS

list of date and time of locations is embedded in table tag, So we will iterate through <td> and extract text.

  1. a) Load html to Cheerio as we do in beautifulsoup

In Python

soup = BeautifulSoup(html,'html5lib')

 

In Cheerio JS

$ = cheerio.load(html);

 

  1. b) This line finds first tr tag in table tag.

var htmlTime = $("table").find('tr');

 

  1. c) Iterate through td tags data by using each() function. This function acts as loop (in Python) iterating through list of elements in which data will be extracted.

htmlTime.each(function (index, element) {      

  // in python, we will use loop, `for element from elements:`

  tag = $(element).find("td");    // in python, `tag = soup.find_all('td')`

  if( tag.text() != "") {

    .

    .

    //EXTRACT DATA

    .

    .

  } else {

    //go to next td tag

    tag = tag.next();

  }

}

 

  1. d) To extract data

Cheerio JS loads html and uses DOM model traverse through. DOM model considers html is tree. So, go to the tag, and scrape data you want.

//extract location(text) enclosed in tag

location = tag.text();

//go to next tag

tag = tag.next();

//extract time(text) enclosed in tag

time = tag.text();

//save in dictionary like in python

loc_list["location"] = location;

loc_list["time"] = time;

 

Some other useful functions:-

1) $(selector, [context], [root])

returns object of selector(any tag) with class or id inside root

2) $(“table”).attr(name, value)

To get tag object having attribute having `value`

3) obj.html()

To get html enclosed in tags

For more just drop in here

Step#3: Execute scraper using command

node <scrapername>.js

 

Hoping that this blog is able to  how to scrape in Javascript by finding similarities with Python.

Resources:

Continue ReadingCreate Scraper in Javascript for Loklak Scraper JS
Read more about the article Best Practices when writing Tests for loklak Server

Best Practices when writing Tests for loklak Server

Why do we write unit-tests? We write them to ensure that developers’ implementation doesn’t change the behaviour of parts of the project. If there is a change in the behaviour, unit-tests throw errors. This keep developers in ease during integration of the software and ensure lower chances of unexpected bugs.

After setting up the tests in Loklak Server, we were able to check whether there is any error or not in the test. Test failures didn’t mention the error and the exact test case at which they failed. It was YoutubeScraperTest that brought some of the best practices in the project. We modified the tests according to it.

The following are some of the best practices in 5 points that we shall follow while writing unit tests:

  1. Assert the assertions

There are many assert methods which we can use like assertNull, assertEquals etc. But we should use one which describes the error well (being more descriptive) so that developer’s effort is reduced while debugging.

Using these assertions related preferences help in getting to the exact errors on test fails, thus helping in easier debugging of the code.

Some examples can be:-

  • Using assertThat() over assertTrue

assertThat() give more descriptive errors over assertTrue(). Like:-

When assertTrue() is used:

java.lang.AssertionError: Expected: is <true> but: was <false> at org.loklak.harvester.TwitterScraperTest.testSimpleSearch(TwitterScraperTest.java:142) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at org.hamcr..........

 

When assertThat() is used:

java.lang.AssertionError:
Expected: is <true>
     but: was <false>
at org.loklak.harvester.TwitterScraperTest.testSimpleSearch(TwitterScraperTest.java:142)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at org.hamcr...........

 

NOTE:- In many cases, assertThat() is preferred over other assert method (read this), but in some cases other methods are used to give better descriptive output (like in next examples)

  • Using assertEquals() over assertThat()

For assertThat()

java.lang.AssertionError:

Expected: is "ar photo #test #car https://pic.twitter.com/vd1itvy8Mx"

but: was "car photo #test #car https://pic.twitter.com/vd1itvy8Mx"

at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)

at org.junit.Assert.assertThat(Ass........

 

For assertEquals()

org.junit.ComparisonFailure: expected:<[c]ar photo #test #car ...> but was:<[]ar photo #test #car ...>

at org.junit.Assert.assertEquals(Assert.java:115)

at org.junit.Assert.assertEquals(Assert.java:144)

at org.loklak.harvester.Twitter.........

 

We can clearly see that second example gives better error description than the first one.(An SO link)

  1. One Test per Behaviour

Each test shall be independent of other with none having mutual dependencies. It shall test only a specific behaviour of the module that is tested.

Have a look of this snippet. This test checks the method that creates the twitter url by comparing the output url method with the expected output url.

@Test

public void testPrepareSearchURL() {

    String url;

    String[] query = {

        "fossasia", "from:loklak_test",

        "spacex since:2017-04-03 until:2017-04-05"

    };

    String[] filter = {"video", "image", "video,image", "abc,video"};

    String[] out_url = {

        "https://twitter.com/search?f=tweets&vertical=default&q=fossasia&src=typd",

        "https://twitter.com/search?f=tweets&vertical=default&q=fossasia&src=typd",

    };

    // checking simple urls

    for (int i = 0; i < query.length; i++) {

        url = TwitterScraper.prepareSearchURL(query[i], "");


        //compare urls with urls created

        assertThat(out_url[i], is(url));

    }

}

 

This unit-test tests whether the method-under-test is able to create twitter link according to query or not.

  1. Selecting test cases for the test

We shall remember that testing is a very costly task in terms of processing. It takes time to execute. That is why, we need to keep the test cases precise and limited. In loklak server, most of the tests are based on connection to the respective websites and this step is very costly. That is why, in implementation, we must use least number of test cases so that all possible corner cases are covered.

  1. Test names

Descriptive test names that are short but give hint about their task which are very helpful. A comment describing what it does is a plus point. The following example is from YoutubeScraperTest. I added this point to my ‘best practices queue’ after reviewing the code (when this module was in review process).

/**

* When try parse video from input stream should check that video parsed.

* @throws IOException if some problem with open stream for reading data.

*/

@Test

public void whenTryParseVideoFromInputStreamShouldCheckThatJSONObjectGood() throws IOException {

    //Some tests related to method

}

 

AND the last one, accessing methods

This point shall be kept in mind. In loklak server, there are some tests that use Reflection API to access private and protected methods. This is the best example for reflection API.

In general, such changes to access specifiers are not allowed, that is why we shall resolve this issue with the help of:-

  •  Setters and Getters (if available, use it or else create them)
  •  Else use Reflection

If the getter methods are not available, using Reflection API will be the last resort to access the private and protected members of the class. Hereunder is a simple example of how a private method can be accessed using Reflection:

void getPrivateMethod() throws Exception {

    A ret = new A();

    Class<?> clazz = ret.getClass();

    Method method = clazz.getDeclaredMethod("changeValue", Integer.TYPE);

    method.setAccessible(true);

    System.out.println(method.invoke(ret, 2)); 
    //set null if method is static

}

 

I should end here. Try applying these practices, go through the links and get sync with these ‘Best Practices’ 🙂

Resources:

Continue ReadingBest Practices when writing Tests for loklak Server
Read more about the article Communication by pySerial python module in PSLab

Communication by pySerial python module in PSLab

In the PSLab Desktop App we use Python for communication between the PC and PSLab device. The PSLab device is connected to PC via USB cable. The power for the hardware device is provided by the host through USB which in this case is a PC. We need well structured methods to establish communication between PC and PSLab device and this is where pySerial module comes in. We will discuss how to communicate efficiently from PC to a device like PSLab itself using pySerial module.

How to read and write data back to PSLab device?

pySerial is a python module which is used to communicate serially with microcontroller devices like Arduino, RaspBerry Pi, PSLab (Pocket Science Lab), etc. Serial data transfer is easier using this module, you just need to open a port and obtain serial object, which provides useful and powerful functionality. Users can send string (which is an array of bytes) or any other data type all data types can be expressed as byte string using struct module in python, read a specific number of bytes or read till some specific character like ‘\n’ is encountered. We are using this module to create custom read and write functions.

How to Install pySerial and obtain serial object for communication?

You can install pySerial using pip by following command

pip install pyserial

Once it’s installed we can now import it in our python script for use.

Obtain Serial Object

In Linux

>>> import serial
>>> ser = serial.Serial(‘/dev/ttyUSB0’)

In Windows

>>> ser = serial.Serial()
>>> ser.baudrate = 19200
>>> ser.port = ‘COM1’

Or

>>> ser = serial.Serial(‘COM1’, 19200)

You can specify other properties like timeout, stopbits, etc to Serial constructor.

Complete list of parameters is available here. Now this “ser” is an object of Serial class that provides all the functionalities through its interface. In PSLab we obtain a serial object and implement custom methods to handle communication which isn’t directly provided by pySerial, for example if we need to implement a function to get the version of the PSLab device connected. Inside the version read function we need to send some bytes to the device in order to obtain the version string from device as a byte response.

What goes under the hood?

We send some sequence of bytes to PSLab device, every sequence of bytes corresponds to a unique function which is already written in device’s firmware. Device recognises the function and responses accordingly.

Let’s look at code to understand it better.

ser.write(struct.Struct(‘B’).pack(11))  #  Sends 11 as byte string
ser.write(struct.Struct(‘B’).pack(5))   #  Sends 5 as bytes string
x = ser.readline()                      #  Reads bytes until ‘\n’ is encountered   

To understand packing and unpacking using struct module, you can have a read at my other blog post Packing And Unpacking Data in JAVA in which I discussed packing and unpacking of data as byte strings and touched a bit on How it’s done in Python.  

You can specify how many bytes you want to read like shown in code below, which is showing and example for 100 bytes :

x = ser.read(100)

After your communication is complete you can simply close the port by:

ser.close()

Based on these basic interface methods more complex functions can be written to handle your specific needs. More details one how to implement custom methods is available at python-communication-library of PSLab which uses pySerial for communication between Client and PSLab device.

An example of custom read function is suppose I want to write a function to read an int from the device. int is of 2 bytes as firmware is written in C, so we read 2 bytes from device and unpack them in client side i.e on PC. For more such custom functions refer packet_handler.py of PSLab python communication library.

def getInt(self):
       “””
       reads two bytes from the serial port and
       returns an integer after combining them
       “””
       ss = ser.read(2)  # reading 2 bytes from serial object
       try:
           if len(ss) == 2:
               return CP.ShortInt.unpack(ss)[0]  # unpacking bytes to make int
       except Exception as ex:
           self.raiseException(ex, “Communication Error , Function : get_Int”)

Resources

Continue ReadingCommunication by pySerial python module in PSLab
Read more about the article Using Flask-REST-JSONAPI’s Resource Manager In Open Event API Server

Using Flask-REST-JSONAPI’s Resource Manager In Open Event API Server

For the nextgen Open Event API Server, we are using flask-rest-jsonapi to write all the API endpoints. The flask-rest-jsonapi is based on JSON API 1.0 Specifications for JSON object responses.

In this blog post, I describe how I wrote API schema and endpoints for an already existing database model in the Open Event API Server. Following this blog post, you can learn how to write similar classes for your database models. Let’s dive into how the API Schema is defined for any Resource in the Open Event API Server. Resource, here, is an object based on a database model. It provides a link between the data layer and your logical data abstraction. This ResourceManager has three classes.

  1. Resource List
  2. Resource Detail
  3. Resource Relationship

(We’ll take a look at the Speakers API.)
First, we see the already implemented Speaker Model :

class Speaker(db.Model):

   """Speaker model class"""

   __tablename__ = 'speaker'

   id = db.Column(db.Integer, primary_key=True)

   name = db.Column(db.String, nullable=False)

   photo = db.Column(db.String)

   website = db.Column(db.String)

   organisation = db.Column(db.String)

   is_featured = db.Column(db.Boolean, default=False)

   sponsorship_required = db.Column(db.Text)

   def __init__(self,

                name=None,

                photo_url=None,

                website=None,

                organisation=None,

                is_featured=False,

                sponsorship_required=None):

      self.name = name

      self.photo = photo_url

      self.website = website

      self.organisation = organisation

      self.is_featured = is_featured

      self.sponsorship_required = sponsorship_required

 

Here’s the Speaker API Schema:

class SpeakerSchema(Schema):
   class Meta:
       type_ = 'speaker'
       self_view = 'v1.speaker_detail'
       self_view_kwargs = {'id': '<id>'}
   id = fields.Str(dump_only=True)
   name = fields.Str(required=True)
   photo_url = fields.Url(allow_none=True)
   website = fields.Url(allow_none=True)
   organisation = fields.Str(allow_none=True)
   is_featured = fields.Boolean(default=False)
   sponsorship_required = fields.Str(allow_none=True)
class SpeakerList(ResourceList):  
   schema = SpeakerSchema
   data_layer = {'session': db.session,
                 'model': Speaker}
class SpeakerDetail(ResourceDetail):
   schema = SpeakerSchema
   data_layer = {'session': db.session,
                 'model': Speaker}
class SpeakerRelationship(ResourceRelationship):
   schema = SpeakerSchema
   data_layer = {'session': db.session,
                 'model': Speaker}

 

Last piece of code is listing the actual endpoints in __init__ file for flask-rest-jsonapi

api.route(SpeakerList, 'speaker_list', '/events/<int:event_id>/speakers', '/sessions/<int:session_id>/speakers', '/users/<int:user_id>/speakers')
api.route(SpeakerDetail, 'speaker_detail', '/speakers/<int:id>')
api.route(SpeakerRelationship, 'speaker_event', '/speakers/<int:id>/relationships/event')
api.route(SpeakerRelationship, 'speaker_user', '/speakers/<int:id>/relationships/user')
api.route(SpeakerRelationship, 'speaker_session', '/speakers/<int:id>/relationships/sessions')

 

How to write API schema from database model?

Each column of the database model is a field in the API schema. These are marshmallow fields and can be of several data types – String, Integer, Float, DateTime, Url.

Three class definitions follow the Schema class.

  • List:
    SpeakerList class is the basis of endpoints:
api.route(SpeakerList, 'speaker_list', '/events/<int:event_id>/speakers',
          '/sessions/<int:session_id>/speakers',                                                             
          '/users/<int:user_id>/speakers')

This class will contain methods that generate a list of speakers based on the id that is passed in view_kwargs. Let’s say that ‘/sessions/<int:session_id>/speakers’ is requested. As the view_kwargs here contains sesssion_id, the query methods in SpeakerList class will fetch a list of speaker profiles related to  the sessions identified by session_id.

The flask-rest-jsonapi allows GET and POST methods for ResourceList. When using these endpoints for POST, the before_create_object and before_post methods can be written. These methods are overridden from the base ResourceList class in flask-rest-jsonapi/resource.py when they are defined in Speaker’s class.

  • Detail: 

SpeakerDetail class provides these endpoints:

 api.route(SpeakerDetail, 'speaker_detail', '/speakers/<int:id>')

The Resource Detail provides methods to facilitate GET, PATCH and DELETE requests provided for the endpoints. Methods like: before_get_object, before_update_object, after_update_object are derived from ResourceDetail class. The endpoints return an object of the resource based on the view_kwargs in a JSON response.

  • Relationship:

SpeakerRelationship class, as you might have guesses, provides:

api.route(SpeakerRelationship, 'speaker_event', '/speakers/<int:id>/relationships/event')

api.route(SpeakerRelationship, 'speaker_user', '/speakers/<int:id>/relationships/user')

api.route(SpeakerRelationship, 'speaker_session', '/speakers/<int:id>/relationships/sessions')

SpeakerRelationship class provides methods to create, update and delete relationships between the speaker and related resources – events, users and sessions in this case.

The above is a bare bone API schema example. The actual implementation in Open Event Server has lots of helper methods too to cater to our specific needs.

Additional Resources:

Continue ReadingUsing Flask-REST-JSONAPI’s Resource Manager In Open Event API Server