Building SUSI.AI Android App with FDroid

Fdroid is an app store for Free and Open Source Software (FOSS). Building and hosting an app on Fdroid is not an easy process compared to when we host one on Google Play. A certain set of build checks are required to be done prior to making a merge request (which is similar to a pull request in GitHub) in the fdroid-data GitLab repository. SUSI.AI Android app has undergone through all these checks and tests and is now ready for the merge request to be made.

Setting up the fdroid-server and fdroid-data repositories is a separate thing and is fairly easy. Building the app using the tools provided by fdroid is another thing and is the one that causes the most problems. It will involve quite a few steps to get started. Fdroid requires all the apps need to be built using:

$ fdroid build -v -l ai.susi

This will output a set of logs which tell us what went wrong in the builds. The usual one in a first time app is obviously the build is not taking place at all. The reason is our metadata file needs to be changed to initiate a build.

The metadata file is used for the build process and contains all the information about the app. The metadata file for a.susi package was a .yaml file.

Builds:

 – versionName: 1.0.10

   versionCode: 11

   commit: 1ad2fd0e858b1256617e652c6c8ce1b8372473e6

   subdir: app

   gradle:

     – fdroid

This is the metadata reference file’s build section that is used for the build process using the command that was mentioned above.The versionName a nd versionCode is found in the build.gradle file in the app and commit denotes the commit-id of the latest commit that will be checked out and built, subdir shows the subdirectory of the app, here the subdirectory is the app file.

Next is the interesting stuff, since we are using flavors in the app, we have to mention in the gradle the flavor which we are using, in our case we are using the flavor by the name of “fdroid” and by mentioning this we can build only the “fdroid” flavor in the app.

Also when building the app there were many blockers that were faced, the reason for the usual build fails were :

1 actionable task: 1 executed
INFO: Scanning source for common problems…
ERROR: Found usual suspect ‘youtube.*android.*player.*api’ at app/libs/YouTubeAndroidPlayerApi.jar
WARNING: Found JAR file at app/libs/YouTubeAndroidPlayerApi.jar
WARNING: Found possible binary at app/src/main/assets/snowboy/alexa_02092017.umdl
WARNING: Found possible binary at app/src/main/assets/snowboy/common.res
ERROR: Found shared library at app/src/main/jniLibs/arm64-v8a/libsnowboy-detect-android.so
ERROR: Found shared library at app/src/main/jniLibs/armeabi-v7a/libsnowboy-detect-android.so
INFO: Removing gradle-wrapper.jar at gradle/wrapper/gradle-wrapper.jar
ERROR: Could not build app ai.susi: Can‘t build due to 3 errors while scanning
INFO: Finished
INFO: 1 build failed

The reason for these build fails were that fdroid does not allow us to use prebuilt files and any proprietary software if present, the above log indicates the two prebuilt files which should be removed and also the YouTubeAndroidPlayerApi.jar which is proprietary software and hence needs to removed. So, to remove the files that are not used in the fdroid flavor and exclude them in the build process, we have to include the following statements in the build section of the metadata reference file :

   rm:
     – app/src/main/jniLibs/arm64-v8a/libsnowboy-detect-android.so
     – app/src/main/jniLibs/armeabi-v7a/libsnowboy-detect-android.so
     – app/libs/YouTubeAndroidPlayerApi.jar

Once the metadata file is complete we are ready to run the build command once again. If you have properly set the environment in your local PC, build will end successfully assuming there were no Java or any other language syntax errors.

It is worth to mention few other facts which are common to Android software projects. Usually the source code is packed in a folder named “app” inside the repository and this is the common scenario if Android Studio builds up the project from scratch. If this “app” folder is one level below the root, that is “android/app”, the build instructions shown above will throw an error as it cannot find the project files.

The reason for this is as it is mentioned “subdir=app” in the metadata file. Change this to “subdir=android/app” and run the build again. The idea is to direct the build to find where the project files are.

Reference:

  1. Metadata : https://f-droid.org/docs/Build_Metadata_Reference/#Build
  2. Publish an app on fdroid: https://blog.fossasia.org/publish-an-open-source-app-on-fdroid/

Setting up environment to build PSLab Android app using Fdroid Build

Fdroid is a place for open source enthusiasts and developers to host their Free and Open Source Software (FOSS) for free and get more people onboard into their community. In order to host an app in their repository, one has to go through a several steps of builds and tests. This is to ensure that the software provided by them are as quality and safe as they can ever be. They are not allowing proprietary libraries or tools to integrate into any app or they will  be published outside the Fdroid main repository (fdroid-data) so that the users will know what they are downloading.

In a normal Linux computer where we are developing Android apps and have setup Android Studio will not be able to run the build command using:

$ fdroid build -v -l org.fossasia.pslab

The reason behind this is that we have not installed gradle and build tools required by the “fdroid build” because they are not useful in our day today activities for standalone activities. First thing we need to do is, install gradle separately. This will include adding gradle to $PATH as well.

Download the latest gradle version zip file or the version your project is using with the following command. In PSLab Android app, we are using 4.5.1 version and the snippet below include that version.

$ wget https://services.gradle.org/distributions/gradle-4.5.1-bin.zip

Next step is to install this in a local folder. We can select any path we want, but /opt/ folder is generally used in such scenarios.

sudo mkdir /opt/gradle
sudo unzip -d /opt/gradle gradle-4.5.1-bin.zip

Then we can add gradle to our $PATH variable using the following command:

$ export PATH=$PATH:/opt/gradle/gradle-4.5.1/bin

Now we are all set with gradle settings. Next step is to verify that the fdroid server is properly configured and up to date. When you run the build command after setting up the gradle in PC, it will throw an error similar to “failed to find any output apks”. This causes if the installed fdroid server version is old.

Fdroid server is running on python 3 and it will require some additional libraries pre-installed to properly function.

$ sudo apt-get install vagrant virtualbox git python3-certifi python3-libvirt python3-requestbuilder python3-yaml python3-clint python3-vagrant python3-paramiko python3-pyasn1 python3-pyasn1-modules

Once these libraries are installed, remove the previous instance of fdroidserver by using the following command:

$ sudo apt-get remove fdroidserver

Then we can reinstall the latest version of fdroid server from git using the following command:

$ git clone https://gitlab.com/fdroid/fdroidserver.git
export PATH="$PATH:$PWD/fdroidserver"

Now we are all set to do a brand new lint build on our PC to make our app ready to be published in Fdroid repository!

Reference:

  1. Install gradle : https://www.vultr.com/docs/how-to-install-gradle-on-ubuntu-16-10
  2. Gradle versions : https://gradle.org/releases
  3. Setting up Fdroid-server : https://f-droid.org/en/docs/Build_Server_Setup/

Installing fdroidserver : https://gitlab.com/fdroid/fdroiddata/blob/master/README.md#quickstart

Build Button Resolution in Meilix

Meilix Generator is a webapp which has a build button and after clicking the button it triggers the travis of Meilix repository. You can even generate ISO from your phone. But while opening the web app on a phone, we came to see that the build button is not properly visible. Footer of the page hide the build button. This blog shows the way to identify the build button issue and to make it responsive for all screen sizes.

Error Rectification

There are two elements that need correction:

  • Build button
  • Footer

Build Button

<input class="btn btn-info mx-auto btn-block" id="file-upload" value="Build" required="" type="submit">

 

Here build button act as a input submit button.

class="form-group"> class="btn btn-info mx-auto btn-block" id="file-upload" value="Build" required="" type="submit" style="height: 50px; width: 360px; border-radius: 500px;"/>

 

We first added the group to include the button in that form. Then we add the build button property to give the button certain look.

But now also the footer overlap the button. So now we need to work on footer part.

Footer

<footer class="footer">

 

We remove footer class and made a new div tag with the id = “deployment”. Then we set the css for the deployment id to customise its appearance.

     #deployment {
      position: absolute;
      text-align: center;
      width: 100%;
      padding: 10px;
      bottom: 0px;
      border-top: 1px solid #bfbfbf;
      background-color: #f9f9f9;
    }

 

This set the footer and build button appropriate to become responsive for all sizes.

Reference:

HTML form tag

Input type Submit tag

Speeding up the Travis Build to Decrease the Building Time

Meilix is the repository which uses build script to generate community version of lubuntu as LXQT Desktop. It usually takes around 25-26 to build and deploy the ISO as a Github Release on master branch.
Observing the build log we can see that there are most of the packages like debootstrap, squashfs-tool, etc which are being fetch and setup at the time of building which results in increasing build time.

The issue is to decrease the build time supplying the packages from Travis so that when we run build.sh we won’t be required to download them again and thus few minutes will get reduced.
We included list of packages to be pre-downloaded in .travis.yml

include:
  - os: linux
    addons:
      apt:
        sources:
          - ubuntu-toolchain-r-test
        packages:
          - debootstrap
          - genisoimage
          - p7zip-full
          - squashfs-tools
          - ubuntu-dev-tools
          - dpkg-dev
          - debhelper
          - fakeroot
          - devscripts

These are some important packages included in the build.sh  as devtools=”debootstrap genisoimage p7zip-full squashfs-tools ubuntu-dev-tools” which are always fetched, so we included it into .travis.yml and downloaded it before entering into the chroot environment. By specifying those packages in the .travis.yml, Travis provides those packages beforehand into the docker container so it will run our script. Since the scripts also include package then when the script runs apt-get it won’t download those packages again. They are specified outside the chroot environment because they are expected to be at the system the build.sh script is run to get the iso. By this way, we get a sharp decrease in build time as the internet in the Travis CI container is not so fast so the package download time can be avoided. Now the build is taking around 15-16 minutes to build and deploy.

One thing to note that we didn’t remove those packages off build.sh so that build.sh works outside Travis CI as well.

References:
Pull #176 by @abishekvashok
Speeding up Travis Build by Travis CI
Faster Build by atchai.com

Updating of Linux Standard Base (lsb) and Shorten of log of Meilix Build

Updating the Linux Standard Base of the Meilix

Originally Meilix uses the Ubuntu mirror to fetch the Kernel source for the building of the Operating System. Therefore whenever a user type lsb_release -a for fetching the required information, they get the Ubuntu build info. The task is to change the config file to update the Linux Base Information from Ubuntu to Meilix.

lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 17.04
Release: 17.04
Codename: zesty

We need to patch a file in the location meilix-default-settings/etc/lsb-release which contains the information of the lsb release and this will overwrite the original configuration of Meilix.
This is how the lsb-release file looks like now:

DISTRIB_ID=Meilix
DISTRIB_RELEASE=17.04
DISTRIB_CODENAME=meilix
DISTRIB_DESCRIPTION="Meilix 17.04"

We made the required changes in the file as shown above and the output is as follows:

Shorten the log length of Meilix in Travis

We are facing issues while deployment of Meilix ISO in Travis and the error follows is ReadTimeout Error. Example log of one of fail build is:

This error gets solved automatically and the the ISO gets deployment after 1 or 2 restart of the build. But this time the error doesn’t get solved and after several attempts of restart of build, the ISO doesn’t get deployed.

Reason behind the error:

Travis is taking a lot of time to build the ISO. Travis logs are exceeding the time limit.

Proposed solution:

Reduce the time of build or shift to a new CI.
Reduce the log of the build so as to get the log within 9999 lines.

Solution Implemented

The best solution is to reduce the number of lines used in the log and this will also reduce the time of the build.
I tried concealing some command outputs by appending >/dev/null 2>&1 to some of the commands that has long outputs and adding -y to the commands like:

apt-get -qq -y --purge install file-roller unrar

References

Wiki of Linux Standard Base
Linux Foundation lsb
Ubuntu answer to reduce log