Shorten the Travis build time of Meilix

Meilix is a lubuntu based script. It uses Travis to compile and deploy the iso as Github Release. Normally Travis took around 17-21 minutes to build the script and then to deploy the iso on the page. It is quite good that one gets the iso in such a small interval of time but if we would able to decrease this time even more then that will be better.

Meilix script consists of basically 2 tests and 1 deploy:

The idea behind reducing the time of Meilix building is to parallely run the tests which are independent to each other. So here we run the build.sh and aptRepoUpdater.sh parallely to reduce the time upto some extent.
This pictures denotes that both the tests are running parallely and Github Releases is waiting for them to get complete successfully to deploy the iso.

Let’s see the code through which this made possible:

jobs:
  include:
    - script: ./build.sh
    - script: 'if [ "$TRAVIS_PULL_REQUEST" = "false" ]; then bash ./scripts/aptRepoUpdater.sh; fi'
    - stage: GitHub Release
      script: echo "Deploying to GitHub releases ..."

Here we included a job section in which we wrote the test which Travis has to carry out parallely. This will run both the script at the same time and can help to reduce the time.
After both the script run successfully then Github Release the iso.

Here we can see that we are only able to save around 30-40 seconds and that much matters a lot in case if we have more than 1 build going on the same time.

Links to follow:
Travis guide to build stages

Firefox Customization for Meilix

Meilix a lightweight operating system can be easily customized. This article talks about the way one must proceed to customize the configuration of Firefox of Meilix or on its own Linux distro and how to copy the configuration file of Firefox directly to the home folder of the user.
Meilix script contains a pref.js file which is responsible for providing the configuration. This file contains various function through which one can edit them according to its need to get the required configuration of its need.

Let’s see an example:

user_pref("browser.startup.homepage", "http://www.google.com/cse/home?cx=partner-pub-6065445074637525:8941524350");

This line is used to set the browser startup page and it can be edited according to user choice to find the same page whenever he starts his Firefox.

There are several lines too which can be edited to make the required changes.

How does this work?

This is the Mozilla User Preference file and should be placed in the location /home/user_name/.mozilla/firefox/*.default/prefs.js. It actually controls the attributes of Firefox preference and set the command from here to change it.

How to use it?

One can directly go and edit it according to the choice to use it.

How meilix script uses it to change the user preference?

As we can see that .mozilla folder should be under the home directory, therefore we copy the .mozilla to the the skel folder, so that it gets automatically copied to the home location and we would be able to use.
There is also a shell script which comes in handy to implement the browser startup page and the script even copies the configuration file.

1.#!/bin/bash

2.# firefox
3.# http://askubuntu.com/questions/73474/how-to-install-firefox-addon-from-command-line-in-scripts
4.for user_name in `ls /home/`
5.do
  6.preferences_file="`echo /home/$user_name/.mozilla/firefox/*.default/prefs.js`"
  7.if [ -f "$preferences_file" ]
  8.then
    9.echo "user_pref(\"browser.startup.homepage\", \"https://google.com/\");" >> $preferences_file
  10.fi
11.done

This file is taken from here. And it used in Meilix to run the script to set the default startup page in Firefox. This will be taken input from the user end from the Meilix Generator webapp and it will change the line 9 url according to the input given by the user.
On line 3, *.default will set automatically by the script itself, it generated randomly.
After that, the script will copy the prefs.js in its location and it will implement the changes.

Links to follow:

Firefox-preference guide
Firefox-editing-configuration

Generate Requirement File for Python App for Meilix-Generator

Meilix-Generator is based upon Flask (a Python framework) which has several dependencies to fulfill before actually running the app properly. This article will guide you through the way I used it to automatically generate the requirement file for Meilix Generator app so that one doesn’t have to manually type all the requirements.

An app powered by Python always has several dependencies to fulfill to run the app successfully. The app root directory contains a file named as requirements.txt which contains the name of the dependency and their version. There are features ways to generate the requirement file for an app but the one which I will demonstrate is the best one. So I used this idea to generate the requirement file for webapp Meilix Generator.

Ways to get the requirement.txt

The internet has a featured way through which one has just to run a command to get a list of all the different dependencies within an app.

pip freeze > requirements.txt

This way will generate a bunch of dependencies that we not even required.

Why do we really require to generate a requirement file?

Yes, one may even ask that we can even write the dependency in the requirements.txt file. Why do we need a command to generate it?

Since because it will take care of two important things:
1. It will ensure that all the dependencies have been included, from user input one may forget to find some of the dependency and to include that.

  1. It will also take care of the Python Package Version Pinning which is really important. People use to version pinning for Python requirements as “>=” style. It’s important to follow “==” style because If we want to install the program in one year in the future, the required packages should be pinned to assure that the API changes in the installed packages do not break the program. Please read here for more info.

The way mentioned below will ensure to provide both these features.

How I generated it for Meilix Generator?

Meilix Generator run on Flask that require a requirement.txt file to fulfill the dependencies. Let’s get straight to the way to generate it for the project.

First we will simply create a requirements.in file in which we will simply mention all the dependencies in a simple way:

Flask
gunicorn
Werkzeug

Now we will use a command to latest packages:

pip install --upgrade -r requirements.in

#Note that if you would like to change the requirements, please edit the requirements.in file and run this command to update the dependencies

Then type this command to generate the requirements.txt file from requirements.in

pip-compile --output-file requirements.txt requirements.in

#fix the versions that definitely work for an eternity.
This will generate a file something as:

click==6.7                # via flask
Flask==0.12.2
gunicorn==19.7.1
itsdangerous==0.24        # via flask
Jinja2==2.9.6             # via flask
MarkupSafe==1.0           # via jinja2
Werkzeug==0.12.2          # via flask

Now you generated a perfect requirements.txt file with all the dependencies satisfied with proper python package pinning.

The meilix-generator repo which uses this:
https://github.com/fossasia/meilix-generator

Heroku Deployment through Travis for Meilix-Generator

This article will tell the way to deploy the Meilix Generator on Heroku with the help of Travis. A successful deployment will help as a test for a good PR. Later in the article, we’ll see the one-button deployment on Heroku.

We will here deploy Meilix Generator on Heroku. The way to deploy the project on Heroku is that one should connect its Github account and deploy it on Heroku. The problem arises when one wants to deploy the project on each and every commit. This will help to test that the commits are passing or not. Here we will see that how to use Travis to deploy on Heroku on each and every commit. If the Travis test passed which means that the changes made in the commit are implemented.
We used the same idea to test the commits for Meilix Generator.

Idea behind it

Travis (.travis.yml) will be helpful to us to achieve this. We will use this deploy build to Heroku on each commit. If it gets successfully deployed then it proves that the commit made is working.

How to implement it

I will use Meilix Generator repository to tell the way to implement this. It is as simple as editing the .travis.yml file. We just have to add few lines to .travis.yml and hence it will get deployed.

deploy:
  provider: heroku
  api_key:
    secure: "YOUR ENCRYPTED API KEY"     # explained below
  app: meilix-generator                    # write the name of the app
  on:
    repo: fossasia/meilix-generator                # repo name
    branch: master                        # branch name

Way to generate the api key:
This is really a matter of concern since if this gets wrong then the deployment will not occur.

Steps:

  • cd into the repository which you want to deploy on Heroku.
  • Login Heroku CI and Travis CI into your terminal and type the following.

travis encrypt $(heroku auth:token) --add deploy.api_key

This will automatically provide the key inside the .travis.yml file.

You can also configure manually using

travis heroku setup

That it, you are done, test the build.

Things are still left:

But we are still left with the test of the PR.
For this we have to create a new app.json file as:

{
    "name": "Meilix-Generator",
    "description": "A webapp which generates iso for you",
    "repository": "https://github.com/fossasia/meilix-generator/",
    "logo": "https://github.com/fossasia/meilix-generator/blob/master/static/logo.png",
    "keywords": [
        "meilix-generator",
        "fossasia",
        "flask"
    ],
    "env": {
        "APP_SECRET_TOKEN": {
            "generator": "secret"
        },
        "ON_HEROKU": "true",
        "FORCE_SSL": "true",
        "INVITATION_CODE": {
            "generator": "secret"
        }
    },
    "buildpacks": {
            "url": "heroku/python"        # this is the only place of concern
        }
}

This code should be put in a file in the root of the repo with the name as app.json.
In the buildpacks : the url should be the one which contains the code base language used.

This can be helpful in 2 ways:

  1. Test the commit made and deploy it on Heroku
    2. One-click deployment button which will deploy the app on Heroku
  • Test the deployment through the URL:

https://heroku.com/deploy?template=https://github.com/user_name/repo_name/tree/master

  • Way to add the button:

[![Deploy](https://www.herokucdn.com/deploy/button.svg)](https://heroku.com/deploy)

How can this idea be helpful to a developer

A developer can use this to deploy its app on Heroku and test the commit automatically and view the quality and status of PR too.

Useful repositories and link which uses this:

I have used the same idea in my project. Do have a look:

https://github.com/fossasia/meilix-generator
deployment on Heroku
one-click deployment
app.json file schema   

Building Meilix in Travis using Heroku

Suppose you have to trigger (start) Travis but not through making a commit but through clicking a button on the webapp of the Meilix Generator. Through the webapp of Meilix Generator, we can pass the tag of the build which will be initiated and can also get the build link which is built by Travis.
Heroku is the place where we have deployed our webapp and through a button on the webapp that we used to start the build on the Travis. We have the access to give a tag to the build and with the help of this, we can even predict the URL of the build beforehand. So one can use it for its own personal project in a number of ways. And how I used this feature in Meilix Generator using Meilix script is described below:

How I used this idea

FOSSASIA meilix repository consists the script of a Linux Operating System based on Lubuntu. It uses Travis to build that script to result in a release of an iso file.

Now we thought an idea of building an autonomous system to start this build and get the release and in the meanwhile also make some required changes to the script to get it into the OS. We came up with an idea of a webapp which ask user its email id and tag of the build and till now a picture from the user which will be set as a wallpaper. It means the user would be able to config its distro according to its need through the graphical interface without a single line to code from the user end.

Through the webapp, a build button is taken as an input to go to a build page which triggers the Travis with the same user configuration to build the iso and deploy it on Github page. The user gets the link to the build on the next page only.

How I implemented this idea

Thanks to Travis API without which our idea is impossible to implement. We used a shell script to outframe our idea. The script takes the input of the user’s, repository, and branch to decide to where the trigger to take place.

There are two files one as travis_token as:

fossasia meilix master    # in the format of user repo branch

And script.sh as:  

#!/bin/bash
cat travis_tokens | while read line;     # this lines takes input of the user, repo and branch
do
    array=(${line})
    user="${array[0]}"
    project="${array[1]}"
    len=${#array[@]}
    for ((i=2; i<len; i++)); do
        branch="${array[i]}"            # supplied each value as variable
        body="{\"request\":{
            \"branch\":\"${branch}\",
            \"config\":{
                \"env\":{
                    \"email\":\"${email}\",    # supplied email and travis tag as environment variable
                    \"TRAVIS_TAG\":\"${TRAVIS_TAG}\"
                }
            }
    }}"
    echo "This Link Will be ready in approx 20 minutes"
    echo "https://github.com/fossasia/meilix/releases/download/${TRAVIS_TAG}/meilix-zesty-`date +%Y%m%d`-i386.iso"                  # a pre-predication of the link, we provide tag from user and date from system.
        curl -s -X POST \           # sending an API POST request to Travis to trigger the build of most recent commit 
            -H "Content-Type: application/json" \
            -H "Accept: application/json" \
            -H "Travis-API-Version: 3" \
            -H "Authorization: token ${KEY}" \     # this is stored in Heroku as KEY as environment variable and supplied from there only
            -d "${body}" \
            "https://api.travis-ci.org/repo/${user}%2F${project}/requests"  #%2 is used to interpret user and repo name as a single URL segment.
    done
done

After the trigger, you will get email which consists of a downloadable link to the iso.

How can this idea be helpful to a developer

There are lots of ways a developer can use this idea out. If a developer wants their user to automatically trigger the build and get the release build directly.

One can use it to set even the commit message through the shell script and customizing build configuration like replace, merge or deep_merge a configuration with the original .travis.yml file present in source repo.

Useful repositories and link which uses this:

Know more about Travis API v3:
Triggering the build
API blog

Have a look at our webapp and generate your own iso:
https://melix-generator.herokuapp.com/

Source code here:
https://github.com/fossasia/meilix-generator
https://github.com/fossasia/meilix

Building the Meilix Generator with Flask

Meilix Generator is a webapp which is used to trigger the Travis build of Meilix and mail the user the link of the iso. Meilix Generator webapp is based on Flask. This blog shows that how easy is to build a webapp and take the HTML files to render it into the webapp as well as to call and pass various function. Here I used Flask, the Python framework to render the HTML templates and send requests for various purposes (mentioned later in the article) without coding everything from scratch because of import facility of the Flask.

What is Flask?

Flask is a Python micro web framework based on Werkzeug, Jinja 2 template engine. It is used as the backbone of the webapp. It features us with a whole set of Python from which we can easily generate webapp. It is micro as it has no tools and no library itself. It come up with minimum requirements and one who needs can import different library and use it. And I used several import function for Meilix Generator like render_template, send_from_directory, etc.

Implementation (The use case in Meilix Generator)

First of all, the installation process: We will do the installation in a virtual environment. We prefer virtual environment to differentiate the Python working environment since few programs are there which require different Python versions to work.
Install virtual environment 

sudo pip install virtualenv

Now go to the folder (project) and activate it using

. venv/bin/activate

Now install Flask

pip install flask
Creating your project

Now it’s time to create a simple project in the directory.
Let’s use HTML as the frontend. In the folder create styles.css for styling and index.html template for the frontend of the page.We will make one app.py file which would look similar to this: 

from flask import Flask, render_template
app = Flask(__name__)
@app.route('/')
def index():
	"""Index page"""
	return render_template("index.html")
if __name__ == '__main__':
    app.run()

Flask looks for the / (root) path and here the root return the main template (index.html) which is the main function.

Compiling it to view the page:

export FLASK_DEBUG=1 FLASK_APP=app.py
flask run

You will find your page at http://127.0.0.1:5000

More options (how more it can help you)

  • Add more HTML template options and refer it in app.py
  • Easily use Github API  from a different .py file (this file should get import to app.py) to fetch data like: https://api.github.com/users/user_name : It will fetch user name, repos, followers and many more important information.

How I used this idea for FOSSASIA (Meilix Generator)

I used Flask for the backbone of project Meilix Generator. First, I used from function to import various library needed for the project and then made several functions for the same. Let’s understand the concept using few example:

from flask import Flask, render_template
@app.route('/about')
def about():
		#About page
		return render_template("about.html")

or

from flask import Flask, send_from_directory
@app.route('/uploads/<filename>')
def uploaded_file(filename):
		return send_from_directory(app.config['UPLOAD_FOLDER'],filename)

For more details file app.py can be found here of the Meilix Generator repository where we used the above idea.

Important Links and Repositories: