Use Travis to extract testing APKs for PSLab

Travis CI is a popular continuous integration tool built to test software and deployments done at GitHub repositories. They provide free plans to open source projects. PSLab Android is using Travis to ensure that all the pull requests made and the merges are build-bug frees. Travis can do this pretty well, but that is not all it can do. It’s a powerful tool and we can think of it as a virtual machine (with high specs) installed in a server for us to utilize efficiently.

There was a need in PSLab development that, if we want to test the functionalities of code at a branch. To fulfil this need, one has to download the branch to his local machine and use gradle to build a new apk. This is a tedious task, not to mention reviewers, even developers wouldn’t like to do this.

If the apk files were generated and in a branch, we can simply download them using a console command.

$ wget https://raw.github.com/fossasia/<repository>/<branch>/<file with extension>

 

With the help of Travis integration, we can create a simple script to generate these apks for us. This is done as follows;

Once the Travis build is complete for a triggering event such as a pull request, it will run it’s “after_success” block. We are going to execute a script at this point. Simply add the following code snippet.

after_success:
  - 'if [ "$TRAVIS_PULL_REQUEST" == "false" ]; then bash <script-name>.sh; fi'

 

This will run the script you have mentioned using bash. Here we will have the following code snippets in the specified bash script.

First of all we have to define the branch we want to build. This can be done using a variable assignment.

export DEVELOPMENT_BRANCH=${DEVELOPMENT_BRANCH:-development}

 

Once the build is complete, there will be new folders in the virtual machine. One of them is the app folder. Inside this folder contains the build folder where all the apk files are generated. So the next step is to copy these apk files to a place of our preference. I am using a folder named apk to make it much sense.

cd apk
\cp -r ../app/build/outputs/apk/*/**.apk .
\cp -r ../app/build/outputs/apk/debug/output.json debug-output.json
\cp -r ../app/build/outputs/apk/release/output.json release-output.json
\cp -r ../README.md .

 

Usually, the build folder has following apk files

  • app-debug.apk
  • app-release-unsigned.apk
  • app-release.apk

Release apks are usually signed with a key and it would cause issues while installation. So we have to filter out the debug apk that we usually use for debugging and get it as the output apk. This involves simple file handling operations in Linux and a bit of git.

First of all, rename the apk file so that it will be different from other files.

# Rename apks with dev prefixes
mv app-debug.apk app-dev-debug.apk

 

Then add and commit them to a specific branch where we want the output from.

git add -A
git commit -am "Travis build pushed [development]"
git push origin apk --force --quiet> /dev/null

 

Once it is all done, you will have a branch created and updated with the apk files you have defined.

 

Figure 1: UI of pslab-android apk branch

Reference:

  1. https://travis-ci.org/
Continue ReadingUse Travis to extract testing APKs for PSLab

Creating an apk in the apk branch using Travis CI

All Android apps in FOSSASIA have an apk branch where after a pull request is merged a new apk gets created so that even people who do not know how to setup the app locally can access it. Let’s see how it is done.

Lets get started

Create a bash file in the scripts folder and name it upload-apk.sh

Here is all the code that you require to create an apk in the apk branch.

First thing that we need to do is setup git. So we’ll set the user name and email just like we do when we setup git for the first time in our own systems.

git config --global user.name "Travis CI"
git config --global user.email "noreply+travis@fossasia.org"

 

Next we will clone the apk branch of the repository and copy all the files that we need ie the apk and the JSON files.

git clone --quiet --branch=apk https://fossasia:$GITHUB_API_KEY@github.com/fossasia/open-event-android apk > /dev/null
cd apk
\cp -r ../app/build/outputs/apk/*/**.apk .
\cp -r ../app/build/outputs/apk/debug/output.json debug-output.json
\cp -r ../app/build/outputs/apk/release/output.json release-output.json

 

Next we will create a new branch that contains only the latest apk. After that we will add the APK and then commit all those changes. You can see that the current date and time are printed out in the commit message.

git checkout --orphan temporary

git add --all .
git commit -am "[Auto] Update Test Apk ($(date +%Y-%m-%d.%H:%M:%S))"

 

We will delete the current apk branch and then rename the current branch to apk. In the end we will force push to origin since histories are unrelated.

git branch -D apk
git branch -m apk

git push origin apk --force --quiet > /dev/null

 

If you have already integrated Travis CI in your repository then you just need to add this line your travis.yml file.

after_success:
- bash scripts/update-apk.sh

 

Now every time a PR gets merged in the repository a new apk file is created in the apk branch of your repository.

Resources

  1. Travis CLI – https://github.com/travis-ci/travis.rb#readme
  2. Travis official documentation – https://travis-ci.org/
Continue ReadingCreating an apk in the apk branch using Travis CI

Automatic Signing and Publishing of Android Apps from Travis

As I discussed about preparing the apps in Play Store for automatic deployment and Google App Signing in previous blogs, in this blog, I’ll talk about how to use Travis Ci to automatically sign and publish the apps using fastlane, as well as how to upload sensitive information like signing keys and publishing JSON to the Open Source repository. This method will be used to publish the following Android Apps:

Current Project Structure

The example project I have used to set up the process has the following structure:

It’s a normal Android Project with some .travis.yml and some additional bash scripts in scripts folder. The update-apk.sh file is standard app build and repo push file found in FOSSASIA projects. The process used to develop it is documented in previous blogs. First, we’ll see how to upload our keys to the repo after encrypting them.

Encrypting keys using Travis

Travis provides a very nice documentation on encrypting files containing sensitive information, but a crucial information is buried below the page. As you’d normally want to upload two things to the repo – the app signing key, and API JSON file for release manager API of Google Play for Fastlane, you can’t do it separately by using standard file encryption command for travis as it will override the previous encrypted file’s secret. In order to do so, you need to create a tarball of all the files that need to be encrypted and encrypt that tar instead. Along with this, before you need to use the file, you’ll have to decrypt in in the travis build and also uncompress it for use.

So, first install Travis CLI tool and login using travis login (You should have right access to the repo and Travis CI in order to encrypt the files for it)

Then add the signing key and fastlane json in the scripts folder. Let’s assume the names of the files are key.jks and fastlane.json

Then, go to scripts folder and run this command to create a tar of these files:

tar cvf secrets.tar fastlane.json key.jks

 

secrets.tar will be created in the folder. Now, run this command to encrypt the file

travis encrypt-file secrets.tar

 

A new file secrets.tar.enc will be created in the folder. Now delete the original files and secrets tar so they do not get added to the repo by mistake. The output log will show the the command for decryption of the file to be added to the .travis.yml file.

Decrypting keys using Travis

But if we add it there, the keys will be decrypted for each commit on each branch. We want it to happen only for master branch as we only require publishing from that branch. So, we’ll create a bash script prep-key.sh for the task with following content

#!/bin/sh
set -e

export DEPLOY_BRANCH=${DEPLOY_BRANCH:-master}

if [ "$TRAVIS_PULL_REQUEST" != "false" -o "$TRAVIS_REPO_SLUG" != "iamareebjamal/android-test-fastlane" -o "$TRAVIS_BRANCH" != "$DEPLOY_BRANCH" ]; then
    echo "We decrypt key only for pushes to the master branch and not PRs. So, skip."
    exit 0
fi

openssl aes-256-cbc -K $encrypted_4dd7_key -iv $encrypted_4dd7_iv -in ./scripts/secrets.tar.enc -out ./scripts/secrets.tar -d
tar xvf ./scripts/secrets.tar -C scripts/

 

Of course, you’ll have to change the commands and arguments according to your need and repo. Specially, the decryption command keys ID

The script checks if the repo and branch are correct, and the commit is not of a PR, then decrypts the file and extracts them in appropriate directory

Before signing the app, you’ll need to store the keystore password, alias and key password in Travis Environment Variables. Once you have done that, you can proceed to signing the app. I’ll assume the variable names to be $STORE_PASS, $ALIAS and $KEY_PASS respectively

Signing App

Now, come to the part in upload-apk.sh script where you have the unsigned release app built. Let’s assume its name is app-release-unsigned.apk.Then run this command to sign it

cp app-release-unsigned.apk app-release-unaligned.apk
jarsigner -verbose -tsa http://timestamp.comodoca.com/rfc3161 -sigalg SHA1withRSA -digestalg SHA1 -keystore ../scripts/key.jks -storepass $STORE_PASS -keypass $KEY_PASS app-release-unaligned.apk $ALIAS

 

Then run this command to zipalign the app

${ANDROID_HOME}/build-tools/25.0.2/zipalign -v -p 4 app-release-unaligned.apk app-release.apk

 

Remember that the build tools version should be the same as the one specified in .travis.yml

This will create an apk named app-release.apk

Publishing App

This is the easiest step. First install fastlane using this command

gem install fastlane

 

Then run this command to publish the app to alpha channel on Play Store

fastlane supply --apk app-release.apk --track alpha --json_key ../scripts/fastlane.json --package_name com.iamareebjamal.fastlane

 

You can always configure the arguments according to your need. Also notice that you have to provide the package name for Fastlane to know which app to update. This can also be stored as an environment variable.

This is all for this blog, you can read more about travis CLI, fastlane features and signing process in these links below:

Continue ReadingAutomatic Signing and Publishing of Android Apps from Travis

Auto Updating SUSI Android APK and App Preview on appetize.io

This blog will cover the way in which the SUSI Android APK is build automatically after each commit and pushed to “apk” branch in the github repo. Other thing which will be covered is that how the app preview on appetize.io can be updated after each commit. This is basically for the testers who wish to test the SUSI Android App. There are four ways to test the SUSI Android App. One is to simply download the alpha version of the app from the Google PlayStore. Here is the link to the app. Join the alpha testing and report bugs on the github issue tracker of the repo. Other way is to build the app from Android Studio but you may need to set the complete project. If you are looking to contribute in the project, this is the advised way to test the app. The other two ways are explained below.

Auto Building of APK and pushing to “apk” branch

We have written a script which does following steps whenever a PR is merged:

  1. Checks if the commit is of a PR or a commit to repo
  2. If not of PR, configures a user whose github account will be used to push the APKs.
  3. Clones the repo, generates the debug and release APK.
  4. Deletes everything in the apk branch.
  5. Commits and Pushes new changes to apk branch.

This script is written for people or testers who do not have android studio installed in their computer and want to test the app. So, they can directly download the apk from the apk branch and install it in their phone. The APK is always updated after each commit. So, whenever a tester downloads the APK from apk branch, he will always get the latest app.

if [[ $CIRCLE_BRANCH != pull* ]]
then
    git config --global user.name "USERNAME"
    git config --global user.email "EMAIL"

    git clone --quiet --branch=apk https://USERNAME:$GITHUB_API_KEY@github.com/fossasia/susi_android apk > /dev/null
    ls
    cp -r ${HOME}/${CIRCLE_PROJECT_REPONAME}/app/build/outputs/apk/app-debug.apk apk/susi-debug.apk
    cp -r ${HOME}/${CIRCLE_PROJECT_REPONAME}/app/build/outputs/apk/app-release-unsigned.apk apk/susi-release.apk
    cd apk

    git checkout --orphan workaround
    git add -A

    git commit -am "[Circle CI] Update Susi Apk"

    git branch -D apk
    git branch -m apk

    git push origin apk --force --quiet > /dev/null
fi

Auto Updating of App Preview on appetize.io

The APKs generated in the above step can now be used to set up the preview of the app on the appetize.io. Appetize.io is an online simulator to run mobile apps ( IOS and Android). Appetize.io provides a nice virtual mobile frame to run native apps with various options like screen size, mobile, OS version, etc. Appetize.io provides some API to update/publish the app. In SUSI, we once uploaded the app on appetize.io and now we are using the API provided by them to update the APK everytime a commit is pushed in the repository.

API information (Derived from official docs of appetize.io):

You may upload a new version of an existing app, or update app settings.

Send an HTTP POST request to

https://APITOKEN@api.appetize.io/v1/apps/PUBLICKEY

Replace APITOKEN with your API token and PUBLICKEY with the public key of the app you’re updating. Your API token must be permissioned to the same account as was used to upload the app. The POST body must be a JSON object. To delete a previously set field, use a value of null.

Optional Fields

  1. url: (string) a publicly accessible link to your .zip, .tar.gz, or .apk file, used to upload a new version of your app.
  2. note: (string) a note for your own purposes, will appear on your management dashboard.

For the url parameter, we have used https://github.com/fossasia/susi_android/raw/apk/susi-debug.apk and note can be anything. We have used Update SUSI Preview.

curl https://$APPETIZE_API_TOKEN@api.appetize.io/v1/apps/mbpprq4xj92c119j7nxdhttjm0 -H 'Content-Type: application/json' -d '{"url":"https://github.com/fossasia/susi_android/raw/apk/susi-debug.apk", "note": "Update SUSI Preview"}'

Summary

This blog covered about how to implement an automatic structure to generate APKs for testing and using that APK to build a preview on websites like appetize.io and then using the APIs provided by them to update the APK after each PR merge in the repo. Check out the resources below to learn more about the topic. So, if you are thinking of contributing to SUSI Android App, this may help you a little in testing the app. But if not, then you can also use the similar technique for your android app as well and ease the life of testers.

Resources

  1. Docs of appetize.io to learn more about the API https://appetize.io/docs
  2. Tutorial on using curl to make API requests https://curl.haxx.se/docs/httpscripting.html
  3. Tutorial on writing basic shell scripts https://ryanstutorials.net/bash-scripting-tutorial/
Continue ReadingAuto Updating SUSI Android APK and App Preview on appetize.io

Enabling Google App Signing for Android Project

Signing key management of Android Apps is a hectic procedure and can grow out of hand rather quickly for large organizations with several independent projects. We, at FOSSASIA also had to face similar difficulties in management of individual keys by project maintainers and wanted to gather all these Android Projects under singular key management platform:

To handle the complexities and security aspect of the process, this year Google announced App Signing optional program where Google takes your existing key’s encrypted file and stores it on their servers and asks you to create a new upload key which will be used to sign further updates of the app. It takes the certificates of your new upload key and maps it to the managed private key. Now, whenever there is a new upload of the app, it’s signing certificate is matched with the upload key certificate and after verification, the app is signed by the original private key on the server itself and delivered to the user. The advantage comes where you lose your key, its password or it is compromised. Before App Signing program, if your key got lost, you had to launch your app under a new package name, losing your existing user base. With Google managing your key, if you lose your upload key, then the account owner can request Google to reassign a new upload key as the private key is secure on their servers.

There is no difference in the delivered app from the previous one as it is still finally signed by the original private key as it was before, except that Google also optimizes the app by splitting it into multiple APKs according to hardware, demographic and other factors, resulting in a much smaller app! This blog will take you through the steps in how to enable the program for existing and new apps. A bit of a warning though, for security reasons, opting in the program is permanent and once you do it, it is not possible to back out, so think it through before committing.

For existing apps:

First you need to go to the particular app’s detail section and then into Release Management > App Releases. There you would see the Get Started button for App Signing.

The account owner must first agree to its terms and conditions and once it’s done, a page like this will be presented with information about app signing infrastructure at top.

So, as per the instructions, download the PEPK jar file to encrypt your private key. For this process, you need to have your existing private key and its alias and password. It is fine if you don’t know the key password but store password is needed to generate the encrypted file. Then execute this command in the terminal as written in Step 2 of your Play console:

java -jar pepk.jar –keystore={{keystore_path}} –alias={{alias}} –output={{encrypted_file_output_path}} –encryptionkey=eb10fe8f7c7c9df715022017b00c6471f8ba8170b13049a11e6c09ffe3056a104a3bbe4ac5a955f4ba4fe93fc8cef27558a3eb9d2a529a2092761fb833b656cd48b9de6a

You will have to change the bold text inside curly braces to the correct keystore path, alias and the output file path you want respectively.

Note: The encryption key has been same for me for 3 different Play Store accounts, but might be different for you. So please confirm in Play console first

When you execute the command, it will ask you for the keystore password, and once you enter it, the encrypted file will be generated on the path you specified. You can upload it using the button on console.

After this, you’ll need to generate a new upload key. You can do this using several methods listed here, but for demonstration we’ll be using command line to do so:

keytool -genkey -v -keystore {{keystore_path}} -alias {{alias_name}} -keyalg RSA -keysize 2048 -validity 10000

The command will ask you a couple of questions related to the passwords and signing information and then the key will be generated. This will be your public key and be used for further signing of your apps. So keep it and the password secure and handy (even if it is expendable now).

After this step, you need to create a PEM upload certificate for this key, and in order to do so, execute this command:

keytool -export -rfc -keystore {{keystore_path}} -alias {{alias_name}} -file {{upload_certificate.pem}}

After this is executed, it’ll ask you the keystore password, and once you enter it, the PEM file will be generated and you will have to upload it to the Play console.

If everything goes right, your Play console will look something like this:

 

Click enrol and you’re done! Now you can go to App Signing section of the Release Management console and see your app signing and new upload key certificates

 

You can use the SHA1 hash to confirm the keys as to which one corresponds to private and upload if ever in confusion.

For new apps:

For new apps, the process is like a walk in park. You just need to enable the App Signing, and you’ll get an option to continue, opt-out or re-use existing key.

 

If you re-use existing key, the process is finished then and there and an existing key is deployed as the upload key for this app. But if you choose to Continue, then App Signing will be enabled and Google will use an arbitrary key as private key for the app and the first app you upload will get its key registered as the upload key

 

This is the screenshot of the App Signing console when there is no first app uploaded and you can see that it still has an app signing certificate of a key which you did not upload or have access to.

If you want to know more about app signing program, check out these links:

Continue ReadingEnabling Google App Signing for Android Project

Preparing for Automatic Publishing of Android Apps in Play Store

I spent this week searching through libraries and services which provide a way to publish built apks directly through API so that the repositories for Android apps can trigger publishing automatically after each push on master branch. The projects to be auto-deployed are:

I had eyes on fastlane for a couple of months and it came out to be the best solution for the task. The tool not only allows publishing of APK files, but also Play Store listings, screenshots, and changelogs. And that is only a subset of its capabilities bundled in a subservice supply.

There is a process before getting started to use this service, which I will go through step by step in this blog. The process is also outlined in the README of the supply project.

Enabling API Access

The first step in the process is to enable API access in your Play Store Developer account if you haven’t done so. For that, you have to open the Play Dev Console and go to Settings > Developer Account > API access.

If this is the first time you are opening it, you’ll be presented with a confirmation dialog detailing about the ramifications of the action and if you agree to do so. Read carefully about the terms and click accept if you agree with them. Once you do, you’ll be presented with a setting panel like this:

Creating Service Account

As you can see there is no registered service account here and we need to create one. So, click on CREATE SERVICE ACCOUNT button and this dialog will pop up giving you the instructions on how to do so:

So, open the highlighted link in the new tab and Google API Console will open up, which will look something like this:

Click on Create Service Account and fill in these details:

Account Name: Any name you want

Role: Project > Service Account Actor

And then, select Furnish a new private key and select JSON. Click CREATE.

A new JSON key will be created and downloaded on your device. Keep this secret as anyone with access to it can at least change play store listings of your apps if not upload new apps in place of existing ones (as they are protected by signing keys).

Granting Access

Now return to the Play Console tab (we were there in Figure 2 at the start of Creating Service Account), and click done as you have created the Service Account now. And you should see the created service account listed like this:

Now click on grant access, choose Release Manager from Role dropdown, and select these PERMISSIONS:

Of course you don’t want the fastlane API to access financial data or manage orders. Other than that it is up to you on what to allow or disallow. Same choice with expiry date as we have left it to never expire. Click on ADD USER and you’ll see the Release Manager created in the user list like below:

Now you are ready to use the fastlane service, or any other release management service for that matter.

Using fastlane

Install fastlane by

sudo gem install fastlane

Go to your project folder and run

fastlane supply init

First it will ask the location of the private key JSON file you downloaded, and then the package name of the application you are trying to initialize fastlane for.

Then it will create metadata folder with listing information excluding the images. So you’ll have to download and place the images manually for the first time

After modifying the listing, images or APK, run the command:

fastlane supply run

That’s it. Your app along with the store listing has been updated!

This is a very brief introduction to the capabilities of the supply service. All interactive options can be supplied via command line arguments, certain parts of the metadata can be omitted and alpha beta management along with release rollout can be done in steps! Make sure to check out the links below:

Continue ReadingPreparing for Automatic Publishing of Android Apps in Play Store

Adding swap space to your DigitalOcean droplet, if you run out of RAM

The Open Event Android App generator runs on a DigitalOcean. The deployment runs on a USD 10 box, that has 1 GB of RAM, but for testing I often use a USD 5 box, that has only 512mb of RAM.

When trying to build an android app using gradle and Java 8, there could be an issue where you run out of RAM (especially if it’s 512 only).

What we can do to remedy this problem is creating a swapfile. On an SSD based system, Swap spaces work almost as fast as RAM, because SSDs have very high R/W speeds.

Check hard disk space availability using

df -h

There should be an output like this

Filesystem      Size  Used Avail Use% Mounted on
udev            238M     0  238M   0% /dev
tmpfs            49M  624K   49M   2% /run
/dev/vda1        20G  1.1G   18G   6% /
tmpfs           245M     0  245M   0% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs           245M     0  245M   0% /sys/fs/cgroup
tmpfs            49M     0   49M   0% /run/user/1001

The steps to create a swap file and allocating it as swap are

sudo fallocate -l 1G /swapfile
sudo chmod 600 /swapfile
sudo mkswap /swapfile
sudo swapon /swapfile

We can verify using

sudo swapon --show
NAME      TYPE  SIZE USED PRIO
/swapfile file 1024M   0B   -1

And now if we see RAM usage using free -h , we’ll see

              total        used        free      shared  buff/cache   available
Mem:           488M         37M         96M        652K        354M        425M
Swap:          1.0G          0B        1.0G

Do not use this as a permanent measure for any SSD based filesystem. It can corrupt your SSD if used as swap for long. We use this only for short periods of time to help us build android apks on low ram systems.

Continue ReadingAdding swap space to your DigitalOcean droplet, if you run out of RAM

Getting Signed Release apk’s from the command line

 

View at Medium.com

If anyone of you has deployed an application on the play store, you may have most probably used Android Studio’s built in Generate signed apkoption.

The generate apk option in android studio

Recently while making the Open Event Apk generator, I had to make release apk’s, so that they could be used by an event organiser to publish their app, plus apk’s had to be signed because if they were not signed, it would be impossible to upload due to checks by Google.

Error shown on the developers console

So since I was building the app using the terminal and I didn’t have the luxury of signing the app using Android studio and I had to look for alternatives. Luckily I found two of them :

  1. Using the Signing configs offered by gradle
  2. Using the Oracle sun jarsigner

First of all the signing configs in gradle is a great way to do this. Most Open source apps use this as a way to put their code out for everyone to view and sucessfully hide any private keys and password.

You just need to add few lines of code in your app level build.gradle file and create a file called keystore.properties

In your keystore.properties, we just need to store the sensitive info and this file will be accessible only to people who are part of the project.

storePassword=myStorePassword
keyPassword=mykeyPassword
keyAlias=myKeyAlias
storeFile=myStoreFileLocation

Next we go to the build.gradle and add these lines to read the keystore.properties file and it’s variables

// Create a variable called keystorePropertiesFile, and initialize it to your
// keystore.properties file, in the rootProject folder.
def keystorePropertiesFile = rootProject.file("keystore.properties")

// Initialize a new Properties() object called keystoreProperties.
def keystoreProperties = new Properties()

// Load your keystore.properties file into the keystoreProperties object.
keystoreProperties.load(new FileInputStream(keystorePropertiesFile))

Next we can add the signingConfigs task and reference the values we got above over there

android {
    signingConfigs {
        config {
            keyAlias keystoreProperties['keyAlias']
            keyPassword keystoreProperties['keyPassword']
            storeFile file(keystoreProperties['storeFile'])
            storePassword keystoreProperties['storePassword']
        }
    }
    ...
  }

So As you see this is as simple as this but according to my requirements this seemed a bit tedious since a person setting up the apk generator had to make a keystore file, then find the build.gradle and change the path of the keystore file according to the server directories. So this does the trick but this can be so tedious for someone with no technical experience, so I researched on other solutions and then I got it : Jarsigner and Zipalign

First of all,the jarsigner and zipalign are 2 great tools and the best part about them is that both of them work perfectly with a just one line commands. For signing the app :

jarsigner -keystore <keystore_file> -storepass <storepassword> <apknameTosigned> <alias>

and then zipaligning :

zipalign -v 4 <unaligned-apk-location> <path-to-generated-aligned-apk>

So this is it, we finally used these 2 commands to sign and zipalign an apk and it works perfectly fine. Please test and share comments of the demo live @ http://192.241.232.231. Ciao !

Continue ReadingGetting Signed Release apk’s from the command line