Use Travis to extract testing APKs for PSLab

Travis CI is a popular continuous integration tool built to test software and deployments done at GitHub repositories. They provide free plans to open source projects. PSLab Android is using Travis to ensure that all the pull requests made and the merges are build-bug frees. Travis can do this pretty well, but that is not all it can do. It’s a powerful tool and we can think of it as a virtual machine (with high specs) installed in a server for us to utilize efficiently.

There was a need in PSLab development that, if we want to test the functionalities of code at a branch. To fulfil this need, one has to download the branch to his local machine and use gradle to build a new apk. This is a tedious task, not to mention reviewers, even developers wouldn’t like to do this.

If the apk files were generated and in a branch, we can simply download them using a console command.

$ wget https://raw.github.com/fossasia/<repository>/<branch>/<file with extension>

 

With the help of Travis integration, we can create a simple script to generate these apks for us. This is done as follows;

Once the Travis build is complete for a triggering event such as a pull request, it will run it’s “after_success” block. We are going to execute a script at this point. Simply add the following code snippet.

after_success:
  - 'if [ "$TRAVIS_PULL_REQUEST" == "false" ]; then bash <script-name>.sh; fi'

 

This will run the script you have mentioned using bash. Here we will have the following code snippets in the specified bash script.

First of all we have to define the branch we want to build. This can be done using a variable assignment.

export DEVELOPMENT_BRANCH=${DEVELOPMENT_BRANCH:-development}

 

Once the build is complete, there will be new folders in the virtual machine. One of them is the app folder. Inside this folder contains the build folder where all the apk files are generated. So the next step is to copy these apk files to a place of our preference. I am using a folder named apk to make it much sense.

cd apk
\cp -r ../app/build/outputs/apk/*/**.apk .
\cp -r ../app/build/outputs/apk/debug/output.json debug-output.json
\cp -r ../app/build/outputs/apk/release/output.json release-output.json
\cp -r ../README.md .

 

Usually, the build folder has following apk files

  • app-debug.apk
  • app-release-unsigned.apk
  • app-release.apk

Release apks are usually signed with a key and it would cause issues while installation. So we have to filter out the debug apk that we usually use for debugging and get it as the output apk. This involves simple file handling operations in Linux and a bit of git.

First of all, rename the apk file so that it will be different from other files.

# Rename apks with dev prefixes
mv app-debug.apk app-dev-debug.apk

 

Then add and commit them to a specific branch where we want the output from.

git add -A
git commit -am "Travis build pushed [development]"
git push origin apk --force --quiet> /dev/null

 

Once it is all done, you will have a branch created and updated with the apk files you have defined.

 

Figure 1: UI of pslab-android apk branch

Reference:

  1. https://travis-ci.org/
Continue Reading

Setting up environment to build PSLab Android app using Fdroid Build

Fdroid is a place for open source enthusiasts and developers to host their Free and Open Source Software (FOSS) for free and get more people onboard into their community. In order to host an app in their repository, one has to go through a several steps of builds and tests. This is to ensure that the software provided by them are as quality and safe as they can ever be. They are not allowing proprietary libraries or tools to integrate into any app or they will  be published outside the Fdroid main repository (fdroid-data) so that the users will know what they are downloading.

In a normal Linux computer where we are developing Android apps and have setup Android Studio will not be able to run the build command using:

$ fdroid build -v -l org.fossasia.pslab

The reason behind this is that we have not installed gradle and build tools required by the “fdroid build” because they are not useful in our day today activities for standalone activities. First thing we need to do is, install gradle separately. This will include adding gradle to $PATH as well.

Download the latest gradle version zip file or the version your project is using with the following command. In PSLab Android app, we are using 4.5.1 version and the snippet below include that version.

$ wget https://services.gradle.org/distributions/gradle-4.5.1-bin.zip

Next step is to install this in a local folder. We can select any path we want, but /opt/ folder is generally used in such scenarios.

sudo mkdir /opt/gradle
sudo unzip -d /opt/gradle gradle-4.5.1-bin.zip

Then we can add gradle to our $PATH variable using the following command:

$ export PATH=$PATH:/opt/gradle/gradle-4.5.1/bin

Now we are all set with gradle settings. Next step is to verify that the fdroid server is properly configured and up to date. When you run the build command after setting up the gradle in PC, it will throw an error similar to “failed to find any output apks”. This causes if the installed fdroid server version is old.

Fdroid server is running on python 3 and it will require some additional libraries pre-installed to properly function.

$ sudo apt-get install vagrant virtualbox git python3-certifi python3-libvirt python3-requestbuilder python3-yaml python3-clint python3-vagrant python3-paramiko python3-pyasn1 python3-pyasn1-modules

Once these libraries are installed, remove the previous instance of fdroidserver by using the following command:

$ sudo apt-get remove fdroidserver

Then we can reinstall the latest version of fdroid server from git using the following command:

$ git clone https://gitlab.com/fdroid/fdroidserver.git
export PATH="$PATH:$PWD/fdroidserver"

Now we are all set to do a brand new lint build on our PC to make our app ready to be published in Fdroid repository!

Reference:

  1. Install gradle : https://www.vultr.com/docs/how-to-install-gradle-on-ubuntu-16-10
  2. Gradle versions : https://gradle.org/releases
  3. Setting up Fdroid-server : https://f-droid.org/en/docs/Build_Server_Setup/

Installing fdroidserver : https://gitlab.com/fdroid/fdroiddata/blob/master/README.md#quickstart

Continue Reading

Adding new test cases for increasing test coverage of Loklak Server

It is a good practice to have test cases covering major portion of actual code base. The idea was same to add new test cases in Loklak Server to increase its test coverage. The results were quite amazing with a significant increase of about 3% in total test coverage of the overall project. And about 80-100% increase in the test coverage of individual files for which tests have been written.

Integration Process

For integration, a total of 6 new test cases have been written:

  1. ASCIITest
  2. GeoLocationTest
  3. CommonPatternTest
  4. InstagramProfileScraperTest
  5. EventBriteCrawlerServiceTest
  6. LocationWiseTimeServiceTest

For increasing code coverage, Java docs have been written which increase the lines of code being covered.

Implementation

Basic implementation of adding a new test case is same. Here’s is an example of EventBriteCrawlerServiceTest implementation. This can be used as a reference for adding a new test case in the project.

Prerequisite: If the test file being written tests any additional external service (e.g. EventBriteCrawlerServiceTest tests any event being created on EventBrite platform) then, a corresponding new test service or test event should be written beforehand on the given platform. For EventBriteCrawlerServiceTest, a test-event has been created.

The given below steps will be followed for creating a new test case (EventBriteCrawlerServiceTest), assuming the prerequisite step has been done:

  • A new java file is generated in test/org/loklak/api/search/ as EventBriteCrawlerServiceTest.java.
  • Define package for the test file and import EventBriteCrawlerService.java file which has to be tested along with necessary packages and methods.

package org.loklak.api.search;

import org.loklak.api.search.EventBriteCrawlerService;
import org.loklak.susi.SusiThought;
import org.junit.Test;
import org.json.JSONArray;
import org.json.JSONObject;
import static org.junit.Assert.assertEquals;
import static org.junit.Assert.assertTrue;
import static org.junit.Assert.assertNotNull;

 

  • Define the class and methods that need to be tested. It is a good idea to break the testing into several test methods rather than testing more features in a single method.

public class EventBriteCrawlerServiceTest {
  @Test
  public void apiPathTest() { }

  @Test
  public void eventBriteEventTest() { }

  @Test
  public void eventBriteOrganizerTest() { }

  @Test
  public void eventExtraDetailsTest() { }
}

 

  • Create an object of EventBriteCrawlerService in each test method.

EventBriteCrawlerService eventBriteCrawlerService = new EventBriteCrawlerService();

 

  • Call specific method of EventBriteCrawlerService that needs to be tested in each of the defined methods of test file and assert the actualresult to the expectedresult.

@Test
public void apiPathTest() {
    EventBriteCrawlerService eventBriteCrawlerService = 
        new EventBriteCrawlerService();

     assertEquals("/api/eventbritecrawler.json",
        eventBriteCrawlerService.getAPIPath());
}

 

  • For methods fetching an actual result (Integration tests) of event page, define and initialise an expected set of results and assert the expected result with the actual result by parsing the json result.

@Test
public void eventBriteOrganizerTest() {
    EventBriteCrawlerService eventBriteCrawlerService =
        new EventBriteCrawlerService();
    String eventTestUrl = "https://www.eventbrite.com/e/
        testevent-tickets-46017636991";
    SusiThought resultPage = eventBriteCrawlerService
        .crawlEventBrite(eventTestUrl);
    JSONArray jsonArray = resultPage.getData();
    JSONObject organizer_details =
        jsonArray.getJSONObject(1);
    String organizer_contact_info = organizer_details
        .getString("organizer_contact_info");
    String organizer_link = organizer_details
        .getString("organizer_link");
    String organizer_profile_link = organizer_details
        .getString("organizer_profile_link");
    String organizer_name = organizer_details
        .getString("organizer_name");

    assertEquals("https://www.eventbrite.com
        /e/testevent-tickets-46017636991
        #lightbox_contact",
        organizer_contact_info);
    assertEquals("https://www.eventbrite.com
        /e/testevent-tickets-46017636991
        #listing-organizer", organizer_link);
    assertEquals("", organizer_profile_link);
    assertEquals("aurabh Srivastava", organizer_name);
}

 

  • If the test file is testing the harvester, then import and add the test class in TestRunner.java file. e.g.

import org.loklak.harvester.TwitterScraperTest;

@RunWith(Suite.class)
@Suite.SuiteClasses({
  TwitterScraperTest.class
})

Testing

The last step will be to check if the test case passes with integration of other tests. The gradle will test the test case along with other tests in the project.

./gradlew build

Resources

Continue Reading

Making App Name Configurable for Open Event Organizer App

Open Event Organizer is a client side android application of Open Event API server created for event organizers and entry managers. The application provides a way to configure the app name via environment variable app_name. This allows the user to change the app name just by setting the environment variable app_name to the new name. I will be talking about its implementation in the application in this blog.

Generally, in an android application, the app name is stored as a static string resource and set in the manifest file by referencing to it. In the Organizer application, the app name variable is defined in the app’s gradle file. It is assigned to the value of environment variable app_name and the default value is assigned if the variable is null. The relevant code in the manifest file is:

def app_name = System.getenv('app_name') ?: "eventyay organizer"

app/build.gradle

The default value of app_name is kept, eventyay organizer. This is the app name when the user doesn’t set environment variable app_name. To reference the variable from the gradle file into the manifest, manifestPlaceholders are defined in the gradle’s defaultConfig. It is a map of key value pairs. The relevant code is:

defaultConfig {
   ...
   ...
   manifestPlaceholders = [appName: app_name]
}

app/build.gradle

This makes appName available for use in the app manifest. In the manifest file, app name is assigned to the appName set in the gradle.

<application
   ...
   ...
   android:label="${appName}"

app/src/main/AndroidManifest.xml

By this, the application name is made configurable from the environment variable.

Links:
1. ManifestPlaceholders documentation
2. Stackoverflow answer about getting environment variable in gradle

Continue Reading

Using Travis CI to Generate Sample Apks for Testing in Open Event Android

In the Open Event Android app we were using Travis already to push an apk of the Android app to the apk branch for easy testing after each commit in the repo. A better way to test the dynamic nature of the app would be to use the samples of different events from the Open Event repo to generate an apk for each sample. This could help us identify bugs and inconsistencies in the generator and the Android app easily. In this blog I will be talking about how this was implemented using Travis CI.

What is a CI?

Continuous Integration is a DevOps software development practice where developers regularly push their code changes into a central repository. After the merge automated builds and tests are run on the code that has been pushed. This helps developers to identify bugs in code quite easily. There are many CI’s available such as Travis, Codecov etc.

Now that we are all caught up with let’s dive into the code.

Script for replacing a line in a file (configedit.sh)

The main role of this script would be to replace a line in the config.json file. Why do we need this? This would be used to reconfigure the Api_Link in the config.json file according to our build parameters. If we want the apk for Mozilla All Hands 2017 to be built, I would use this  script to replace the Api_Link in the config.json file to the one for Mozilla All Hands 2017.

This is what the config.json file for the app looks like.

{
   "Email": "[email protected]",
   "App_Name": "Open Event",
   "Api_Link": "https://eventyay.com/api/v1/events/6/"
 }

We are going to replace line 4 of this file with

“Api_Link”:”https://raw.githubusercontent.com/fossasia/open-event/master/sample/MozillaAllHands17

VAR=0
 STRING1=$1
 while read line
 do
 ((VAR+=1))
 if [ "$VAR" = 4 ]; then
 echo "$STRING1"
 else
 echo "$line"
 fi
 done < app/src/main/assets/config.json

The script above reads the file line by line. If it reaches line 4 it would print out the string that was given into the script as a parameter else it just prints out the line in the file.

This script would print the required file for us in the terminal if called but NOT create the required file. So we redirect the output of this file into the same file config.json.

Now let’s move on to the main script which is responsible for the building of the apks.

Build Script(generate_apks.sh)

Main components of the script

  • Build the apk for the default sample i.e FOSSASIA17 using the build scripts ./gradlew build and ./gradlew assembleRelease.
  • Store all the Api_Links and apk names for which we need the apks for different arrays
  • Replace the Api_Link in the json file found under android/app/src/main/assets/config.json using the configedit.sh.
  • Run the build scripts ./gradlew build and ./gradlew assembleRelease to generate the apk.
  • Move the generated apk from app/build/outputs/apk/ to a folder called daily where we store all the generated apks.
  • We then repeat this process for the other Api_Links in the array.

As of now we are generating the apks for the following events:

  1. FOSSASIA 17
  2. Mozilla All Hands 17
  3. Google I/O 17
  4. Facebook Developer Conference 17

Care is also taken to avoid all these builds if it is a PR. All the apks are generated only when there is a commit on the development branch i.e when the PR is merged.

Usage of Scripts in .travis.yml

To add Travis integration for the repo we need to include a file named .travis.yml in the repo which indicates Travis CI what to build.

language: android
 ….
 jdk: oraclejdk8
 ….
 before_script:
   - cd android
 script:
   - chmod +x generate_apks.sh
   - chmod +x configedit.sh
   - ./generate_apks.sh
 …..
 after_success:
   - bash <(curl -s https://codecov.io/bash)
   - cd ..
   - chmod +x upload-apk.sh
   - ./upload-apk.sh

In this file we need to define the language for which Travis will build. Here we indicate that it is android. We also specify the jdk version to be used.

Now let’s talk about the other parts of this snippet.

  • before_script : Executes the bash instructions before the travis build starts. Here we do cd android so that we can access gradlew for building the apk.
  • script : This section consists of the instruction to be executed for the build. Here we give executable rights to the two scripts that we have written sh and . Then ./generate_apks is called and the project build starts. All the apks get saved to the folder daily.
  • after_success : This section consists the instructions that are run after the script executes successfully. Here we see that we run a script called sh. This script is responsible of pushing the generated apk files in an orphan branch called apk.

Some points of Interest

  • If the user/developer testing the apk is in the offline state and then comes online there will be database inconsistencies as data from the local assets as well as the data from the Api_Link would appear in the app.
  • When the app generator CLI is ready we can use it to trigger the builds instead of just replacing the Api_Link. This would also be effective in testing the app generator simultaneously.

Now we have everything setup to trigger builds for various samples after each commit.

Resources

Continue Reading

Using Jackson Library in Open Event Android App for JSON Parsing

Jackson library is a popular library to map Java objects to JSON and vice-versa and has a better parsing speed as compared to other popular parsing libraries like GSON, JSONP etc. This blog post gives a basic explanation on how we utilised the Jackson library in the Open Event Android App.

To use the Jackson library we need to add the dependency for it in the app/build.gradle file.

compile 'com.squareup.retrofit2:converter-jackson:2.2.0'

After updating the app/build.gradle file with the code above we were able to use Jackson library in our project.

Example of Jackson Library JSON Parsing

To explain how the mapping is done let us take an example. The file given below is the sponsors.json file from the android app.

[
    “description”: “”,
    “id”: 1,
    “level”: 3,
    “name”:  KI Group,
    “type”: Gold,
    “url: “”,
    “logo-url”: “” 
]

The sponsors.json consists of mainly 7 attributes namely description, id, level, name, type, url and logo url for describing one sponsor of the event. The Sponsors.java file for converting the json response to Java POJO objects was done as follows utilizing the Jackson library:

public class Sponsors extends RealmObject {
    @JsonProperty(“id”)
    private int id;

    @JsonProperty(“type”)
    private String type;

    @JsonProperty(“description”)
    private String description;

    @JsonProperty(“level”)
    private String level;

    @JsonProperty(“name”)
    private String name;

    @JsonProperty(“url”)
    private String url;

    @JsonProperty(“logo-url”)
    private String logoUrl;
}

As we can see from the above code snippet, the JSON response is converted to Java POJO objects simply by using the annotation “@JsonProperty(“”)” which does the work for us.

Another example which makes this library amazing are the setter and getter annotations which help us use a single variable for two different json attributes if need be. We faced this situation when we were moving from the old api to the new json api. In that case we wanted support for both, old and new json attributes. In that case we simply used the following code snippet which made our transition to new api easier.

public class Sponsors extends RealmObject {
    private int id;
    private String type;
    private String description;
    private String level;
    private String name;
    private String url;
    private String logoUrl;

    @JsonSetter(“logo”)
    public void setLogo(String logoUrl) {
        this.logoUrl = logoUrl;
    }

    @JsonSetter(“logo-url”)
    public void setLogo(String logoUrl) {
        this.logoUrl = logoUrl;
    }
}

As we can see the setter annotations allow easy naming of variable to multiple attributes if need be thus making the code easily adaptable with less overload.

Related Links:

Continue Reading

Flavors in Gradle

Ever wondered how do people maintain different versions of the same app on play store with some customisations in each version. For example, Lite version and a pro version which signify free and paid versions with extra features in paid one.

With the arrival of gradle as a build tool, we got gradle flavors which is an excellent way to have some variations in your app versions.

It can also be leveraged to do hermetic testing with prod/mock flavors. Some other examples could be free/paid flavors and stable/experimental flavors etc.

Now let’s talk about on how to proceed :

The process is very simple

  1. Create the flavors in you build.gradle file
android {  
      productFlavors {  
           mock {   
                applicationIdSuffix = ".mock"  
           }  
           prod  
      }  
 }

Here prod and mock are two flavors and the customisations we add are that mock has a applicationIdSuffix. We can also add altogether different applicationIds as well as define a different version for different flavors and a lot of other different things

2. Now we sync the project

If you now open Build Variants tool window either using the quick access menu located in the status bar in the bottom left hand corner of the Android Studio main window or using the Build Variant tool window bar. Once loaded, clicking in the Build Variant cell for the app module should now list the four build variants: mockDebug, mockRelease and prodDebug, prodRelease

After this we get different folders corresponding to the different flavors. We go ahead and add the different folders where we want the changes to occur. For example if I want to show two different maps for two different versions, where one version is for fdroid and another one is for googleplay and hence we replace google maps with OpenStreet Maps in the fdroid version. Hence, We add res/values and res/layout folders for different resources and we also add different java folders and add the classes that would be different for both the flavors. Since both of these will require different set of permissions we can also add AndroidManifest.xml for these were we define the permissions etc. we want.

Not that we only add the changes for each file. All the other essential things can be added from the Main flavor of the android project.

Any variable available through your code

Another thing to know is that the you can have add variables like this so that you can use different variables for different flavors. For examle if you have different api’s and different options for each flavor like if you want to report crashes in one of the flavors and not in other. Here HOST variable is not the only one you can expose in your code. You can do it with whatever you want:

prod {  
    applicationId "zuul.com.android"
    buildConfigField 'String', 'HOST', '"http://api.zuul.com"'
    buildConfigField 'String', 'FLAVOR', '"prod"'
    buildConfigField "boolean", "REPORT_CRASHES", "true"
}

You can access them as follows:

BuildConfig.HOST  
BuildConfig.FLAVOR  
BuildConfig.REPORT_CRASHES

So I think this is a really useful thing to know and is used in almost every app. We all need some configuration differences in our apps and gradle flavors is the best way to go about it. Go nuts with your imagination on the usage of flavors 😛

Adios!

Continue Reading
Close Menu