Shorten the Travis build time of Meilix

Meilix is a lubuntu based script. It uses Travis to compile and deploy the iso as Github Release. Normally Travis took around 17-21 minutes to build the script and then to deploy the iso on the page. It is quite good that one gets the iso in such a small interval of time but if we would able to decrease this time even more then that will be better.

Meilix script consists of basically 2 tests and 1 deploy:

The idea behind reducing the time of Meilix building is to parallely run the tests which are independent to each other. So here we run the build.sh and aptRepoUpdater.sh parallely to reduce the time upto some extent.
This pictures denotes that both the tests are running parallely and Github Releases is waiting for them to get complete successfully to deploy the iso.

Let’s see the code through which this made possible:

jobs:
  include:
    - script: ./build.sh
    - script: 'if [ "$TRAVIS_PULL_REQUEST" = "false" ]; then bash ./scripts/aptRepoUpdater.sh; fi'
    - stage: GitHub Release
      script: echo "Deploying to GitHub releases ..."

Here we included a job section in which we wrote the test which Travis has to carry out parallely. This will run both the script at the same time and can help to reduce the time.
After both the script run successfully then Github Release the iso.

Here we can see that we are only able to save around 30-40 seconds and that much matters a lot in case if we have more than 1 build going on the same time.

Links to follow:
Travis guide to build stages

Firefox Customization for Meilix

Meilix a lightweight operating system can be easily customized. This article talks about the way one must proceed to customize the configuration of Firefox of Meilix or on its own Linux distro and how to copy the configuration file of Firefox directly to the home folder of the user.
Meilix script contains a pref.js file which is responsible for providing the configuration. This file contains various function through which one can edit them according to its need to get the required configuration of its need.

Let’s see an example:

user_pref("browser.startup.homepage", "http://www.google.com/cse/home?cx=partner-pub-6065445074637525:8941524350");

This line is used to set the browser startup page and it can be edited according to user choice to find the same page whenever he starts his Firefox.

There are several lines too which can be edited to make the required changes.

How does this work?

This is the Mozilla User Preference file and should be placed in the location /home/user_name/.mozilla/firefox/*.default/prefs.js. It actually controls the attributes of Firefox preference and set the command from here to change it.

How to use it?

One can directly go and edit it according to the choice to use it.

How meilix script uses it to change the user preference?

As we can see that .mozilla folder should be under the home directory, therefore we copy the .mozilla to the the skel folder, so that it gets automatically copied to the home location and we would be able to use.
There is also a shell script which comes in handy to implement the browser startup page and the script even copies the configuration file.

1.#!/bin/bash

2.# firefox
3.# http://askubuntu.com/questions/73474/how-to-install-firefox-addon-from-command-line-in-scripts
4.for user_name in `ls /home/`
5.do
  6.preferences_file="`echo /home/$user_name/.mozilla/firefox/*.default/prefs.js`"
  7.if [ -f "$preferences_file" ]
  8.then
    9.echo "user_pref(\"browser.startup.homepage\", \"https://google.com/\");" >> $preferences_file
  10.fi
11.done

This file is taken from here. And it used in Meilix to run the script to set the default startup page in Firefox. This will be taken input from the user end from the Meilix Generator webapp and it will change the line 9 url according to the input given by the user.
On line 3, *.default will set automatically by the script itself, it generated randomly.
After that, the script will copy the prefs.js in its location and it will implement the changes.

Links to follow:

Firefox-preference guide
Firefox-editing-configuration

Generate Requirement File for Python App for Meilix-Generator

Meilix-Generator is based upon Flask (a Python framework) which has several dependencies to fulfill before actually running the app properly. This article will guide you through the way I used it to automatically generate the requirement file for Meilix Generator app so that one doesn’t have to manually type all the requirements.

An app powered by Python always has several dependencies to fulfill to run the app successfully. The app root directory contains a file named as requirements.txt which contains the name of the dependency and their version. There are features ways to generate the requirement file for an app but the one which I will demonstrate is the best one. So I used this idea to generate the requirement file for webapp Meilix Generator.

Ways to get the requirement.txt

The internet has a featured way through which one has just to run a command to get a list of all the different dependencies within an app.

pip freeze > requirements.txt

This way will generate a bunch of dependencies that we not even required.

Why do we really require to generate a requirement file?

Yes, one may even ask that we can even write the dependency in the requirements.txt file. Why do we need a command to generate it?

Since because it will take care of two important things:
1. It will ensure that all the dependencies have been included, from user input one may forget to find some of the dependency and to include that.

  1. It will also take care of the Python Package Version Pinning which is really important. People use to version pinning for Python requirements as “>=” style. It’s important to follow “==” style because If we want to install the program in one year in the future, the required packages should be pinned to assure that the API changes in the installed packages do not break the program. Please read here for more info.

The way mentioned below will ensure to provide both these features.

How I generated it for Meilix Generator?

Meilix Generator run on Flask that require a requirement.txt file to fulfill the dependencies. Let’s get straight to the way to generate it for the project.

First we will simply create a requirements.in file in which we will simply mention all the dependencies in a simple way:

Flask
gunicorn
Werkzeug

Now we will use a command to latest packages:

pip install --upgrade -r requirements.in

#Note that if you would like to change the requirements, please edit the requirements.in file and run this command to update the dependencies

Then type this command to generate the requirements.txt file from requirements.in

pip-compile --output-file requirements.txt requirements.in

#fix the versions that definitely work for an eternity.
This will generate a file something as:

click==6.7                # via flask
Flask==0.12.2
gunicorn==19.7.1
itsdangerous==0.24        # via flask
Jinja2==2.9.6             # via flask
MarkupSafe==1.0           # via jinja2
Werkzeug==0.12.2          # via flask

Now you generated a perfect requirements.txt file with all the dependencies satisfied with proper python package pinning.

The meilix-generator repo which uses this:
https://github.com/fossasia/meilix-generator

Heroku Deployment through Travis for Meilix-Generator

This article will tell the way to deploy the Meilix Generator on Heroku with the help of Travis. A successful deployment will help as a test for a good PR. Later in the article, we’ll see the one-button deployment on Heroku.

We will here deploy Meilix Generator on Heroku. The way to deploy the project on Heroku is that one should connect its Github account and deploy it on Heroku. The problem arises when one wants to deploy the project on each and every commit. This will help to test that the commits are passing or not. Here we will see that how to use Travis to deploy on Heroku on each and every commit. If the Travis test passed which means that the changes made in the commit are implemented.
We used the same idea to test the commits for Meilix Generator.

Idea behind it

Travis (.travis.yml) will be helpful to us to achieve this. We will use this deploy build to Heroku on each commit. If it gets successfully deployed then it proves that the commit made is working.

How to implement it

I will use Meilix Generator repository to tell the way to implement this. It is as simple as editing the .travis.yml file. We just have to add few lines to .travis.yml and hence it will get deployed.

deploy:
  provider: heroku
  api_key:
    secure: "YOUR ENCRYPTED API KEY"     # explained below
  app: meilix-generator                    # write the name of the app
  on:
    repo: fossasia/meilix-generator                # repo name
    branch: master                        # branch name

Way to generate the api key:
This is really a matter of concern since if this gets wrong then the deployment will not occur.

Steps:

  • cd into the repository which you want to deploy on Heroku.
  • Login Heroku CI and Travis CI into your terminal and type the following.

travis encrypt $(heroku auth:token) --add deploy.api_key

This will automatically provide the key inside the .travis.yml file.

You can also configure manually using

travis heroku setup

That it, you are done, test the build.

Things are still left:

But we are still left with the test of the PR.
For this we have to create a new app.json file as:

{
    "name": "Meilix-Generator",
    "description": "A webapp which generates iso for you",
    "repository": "https://github.com/fossasia/meilix-generator/",
    "logo": "https://github.com/fossasia/meilix-generator/blob/master/static/logo.png",
    "keywords": [
        "meilix-generator",
        "fossasia",
        "flask"
    ],
    "env": {
        "APP_SECRET_TOKEN": {
            "generator": "secret"
        },
        "ON_HEROKU": "true",
        "FORCE_SSL": "true",
        "INVITATION_CODE": {
            "generator": "secret"
        }
    },
    "buildpacks": {
            "url": "heroku/python"        # this is the only place of concern
        }
}

This code should be put in a file in the root of the repo with the name as app.json.
In the buildpacks : the url should be the one which contains the code base language used.

This can be helpful in 2 ways:

  1. Test the commit made and deploy it on Heroku
    2. One-click deployment button which will deploy the app on Heroku
  • Test the deployment through the URL:

https://heroku.com/deploy?template=https://github.com/user_name/repo_name/tree/master

  • Way to add the button:

[![Deploy](https://www.herokucdn.com/deploy/button.svg)](https://heroku.com/deploy)

How can this idea be helpful to a developer

A developer can use this to deploy its app on Heroku and test the commit automatically and view the quality and status of PR too.

Useful repositories and link which uses this:

I have used the same idea in my project. Do have a look:

https://github.com/fossasia/meilix-generator
deployment on Heroku
one-click deployment
app.json file schema   

Customizing Firefox in Meilix using skel

We had a problem of customizing the Firefox browser configuration using Meilix generator so we used the skel folder in Linux. The /etc/skel directory contains files and directories that are automatically copied over to a new user’s home directory when such user is created by the useradd program.

Firefox generally reads its settings from ~/.mozilla/firefox folder present in Linux home of the user, which is changed when user modifies his settings like changing the homepage or disabling the bookmarks. To add these changes for every new user we can copy these configurations into skel folder.

Which we have done with Meilix which have these configurations in it and can be modified by using Meilix generator.

If you are going to look in your ~/.mozilla/firefox folder you will find a random string followed by .default folder this will contain the default settings of your Firefox browser and this random string is created uniquely for every user. The profile name will be different for all users but should always end with .default so if we use skel the .default will be same for everyone and we may choose any name for that like we have used meilix.default in below script.

For example to change the default homepage URL we can create a script that can be used by Meilix generator to modify the URL in Meilix.

#!/bin/bash

set -eu   	 
# firefox
preferences_file="`echo meilix-default-settings/etc/skel/.mozilla/firefox/meilix.default/user.js`"
if [ -f "$preferences_file" ]
then
	echo "user_pref(\"browser.startup.homepage\", \"${event_url}\");" >> $preferences_file
fi

 

Here the ${event_url} is the environment variable sent by Meilix generator to be added to configuration of Firefox that is added to skel folder in Meilix.

We can similarly edit more configuration by editing the file pref.js file it contains the configuration for Firefox like bookmarks, download location, default URL etc.

Resources

Setting Environment Variables up in Travis and Heroku for Meilix and Meilix-Generator

Meilix Generator is a webapp whose task is to take input as a configuration and start the Meilix build. But an anonymous person cannot start the Meilix build of any user and deploy the release in the repository. There are ways which are used as authentication passes through environment variable to start the build. In this article, I show the way I used to trigger Meilix by setting up environment variables in Meilix Generator.

Environment variables are of great use when one has to supply personal token in an open-source project for accessing the repository. So down there, we will have ways to configure the variables in Heroku and Travis. There are so many wikis out there but this one is the blend of both Heroku and Travis.

Heroku

There are several ways to setup variables in Heroku. The way I’m going to describe below is used to access Travis build using Heroku.
Using the Heroku variable generated using Travis will help to trigger the build on Travis.

How:

Idea:
We will use Travis CLI to generate a token (unique and keep it secret). Then provide the token as a variable name to the Heroku.
Backdoor:
This Travis token will give access to the Heroku to trigger the build on that particular Travis account. We use variable to provide the token since in the script we will use this variable as an environment variable to fetch the token in the place of token like as $token.
Implementation:
Open your terminal and type the following:

sudo apt install ruby ruby-dev
sudo gem install travis                       	# install Travis CLI
travis login --org   					# login into Travis
travis token --org		# generate your secret personal access token

You will get a token, copy and paste it into your Heroku app’s settings config vars token. You have to use the `KEY` as the variable which is used in the script for triggering the build. Save it and you are done the setting of the token in the Heroku.

Travis

Now it’s time for Travis token.
It is used to deploy the build to that repository only.
We can use the token in two ways either paste it in the setting of that repository on Travis or pasting the encrypted form of that in the .travis.yml file in that repository. Both will work. But one thing to remember that you must have the write access to that repository.

How

,Idea:
It is used in .travis.yml file as an environment variable to successfully build and deploy the application as a Github release.
Backdoor:
The token gives the permission to Travis to deploy the build application in the GitHub release of the repo(if one using to deploy it there only).
Implementation:
Head up to Github and generate a personal access token with scope repo. Copy the generated token in a safe place.

Way 1:

Paste the token in the setting of the repo in Travis in Environment Variable option. Now it will access the Github repository since it has got the permission from the personal token generated from Github.

Way 2:
Open terminal:

cd repo_name				# cd into the cloned repo
travis encrypt secret_token	#replace secret_token with the token generated

or

travis encrypt secret_token -r user/repo 	#if you are not in the repo

Copy that encrypted token and paste it in proper format in the .travis.yml file. Now you enabled Travis giving permission to release the build.

How can this idea be helpful to a developer

A developer can use this to build the Github Release in its repository. One can secure its token using this technique. One can use it to trigger its personal project in Travis using Heroku.

Useful repositories and link which uses this:

I have used the same idea in my project. Do have a look https://github.com/fossasia/meilix-generator
about environment variable
encryption keys
triggering build

Building Meilix in Travis using Heroku

Suppose you have to trigger (start) Travis but not through making a commit but through clicking a button on the webapp of the Meilix Generator. Through the webapp of Meilix Generator, we can pass the tag of the build which will be initiated and can also get the build link which is built by Travis.
Heroku is the place where we have deployed our webapp and through a button on the webapp that we used to start the build on the Travis. We have the access to give a tag to the build and with the help of this, we can even predict the URL of the build beforehand. So one can use it for its own personal project in a number of ways. And how I used this feature in Meilix Generator using Meilix script is described below:

How I used this idea

FOSSASIA meilix repository consists the script of a Linux Operating System based on Lubuntu. It uses Travis to build that script to result in a release of an iso file.

Now we thought an idea of building an autonomous system to start this build and get the release and in the meanwhile also make some required changes to the script to get it into the OS. We came up with an idea of a webapp which ask user its email id and tag of the build and till now a picture from the user which will be set as a wallpaper. It means the user would be able to config its distro according to its need through the graphical interface without a single line to code from the user end.

Through the webapp, a build button is taken as an input to go to a build page which triggers the Travis with the same user configuration to build the iso and deploy it on Github page. The user gets the link to the build on the next page only.

How I implemented this idea

Thanks to Travis API without which our idea is impossible to implement. We used a shell script to outframe our idea. The script takes the input of the user’s, repository, and branch to decide to where the trigger to take place.

There are two files one as travis_token as:

fossasia meilix master    # in the format of user repo branch

And script.sh as:  

#!/bin/bash
cat travis_tokens | while read line;     # this lines takes input of the user, repo and branch
do
    array=(${line})
    user="${array[0]}"
    project="${array[1]}"
    len=${#array[@]}
    for ((i=2; i<len; i++)); do
        branch="${array[i]}"            # supplied each value as variable
        body="{\"request\":{
            \"branch\":\"${branch}\",
            \"config\":{
                \"env\":{
                    \"email\":\"${email}\",    # supplied email and travis tag as environment variable
                    \"TRAVIS_TAG\":\"${TRAVIS_TAG}\"
                }
            }
    }}"
    echo "This Link Will be ready in approx 20 minutes"
    echo "https://github.com/fossasia/meilix/releases/download/${TRAVIS_TAG}/meilix-zesty-`date +%Y%m%d`-i386.iso"                  # a pre-predication of the link, we provide tag from user and date from system.
        curl -s -X POST \           # sending an API POST request to Travis to trigger the build of most recent commit 
            -H "Content-Type: application/json" \
            -H "Accept: application/json" \
            -H "Travis-API-Version: 3" \
            -H "Authorization: token ${KEY}" \     # this is stored in Heroku as KEY as environment variable and supplied from there only
            -d "${body}" \
            "https://api.travis-ci.org/repo/${user}%2F${project}/requests"  #%2 is used to interpret user and repo name as a single URL segment.
    done
done

After the trigger, you will get email which consists of a downloadable link to the iso.

How can this idea be helpful to a developer

There are lots of ways a developer can use this idea out. If a developer wants their user to automatically trigger the build and get the release build directly.

One can use it to set even the commit message through the shell script and customizing build configuration like replace, merge or deep_merge a configuration with the original .travis.yml file present in source repo.

Useful repositories and link which uses this:

Know more about Travis API v3:
Triggering the build
API blog

Have a look at our webapp and generate your own iso:
https://melix-generator.herokuapp.com/

Source code here:
https://github.com/fossasia/meilix-generator
https://github.com/fossasia/meilix

Building the Meilix Generator with Flask

Meilix Generator is a webapp which is used to trigger the Travis build of Meilix and mail the user the link of the iso. Meilix Generator webapp is based on Flask. This blog shows that how easy is to build a webapp and take the HTML files to render it into the webapp as well as to call and pass various function. Here I used Flask, the Python framework to render the HTML templates and send requests for various purposes (mentioned later in the article) without coding everything from scratch because of import facility of the Flask.

What is Flask?

Flask is a Python micro web framework based on Werkzeug, Jinja 2 template engine. It is used as the backbone of the webapp. It features us with a whole set of Python from which we can easily generate webapp. It is micro as it has no tools and no library itself. It come up with minimum requirements and one who needs can import different library and use it. And I used several import function for Meilix Generator like render_template, send_from_directory, etc.

Implementation (The use case in Meilix Generator)

First of all, the installation process: We will do the installation in a virtual environment. We prefer virtual environment to differentiate the Python working environment since few programs are there which require different Python versions to work.
Install virtual environment 

sudo pip install virtualenv

Now go to the folder (project) and activate it using

. venv/bin/activate

Now install Flask

pip install flask
Creating your project

Now it’s time to create a simple project in the directory.
Let’s use HTML as the frontend. In the folder create styles.css for styling and index.html template for the frontend of the page.We will make one app.py file which would look similar to this: 

from flask import Flask, render_template
app = Flask(__name__)
@app.route('/')
def index():
	"""Index page"""
	return render_template("index.html")
if __name__ == '__main__':
    app.run()

Flask looks for the / (root) path and here the root return the main template (index.html) which is the main function.

Compiling it to view the page:

export FLASK_DEBUG=1 FLASK_APP=app.py
flask run

You will find your page at http://127.0.0.1:5000

More options (how more it can help you)

  • Add more HTML template options and refer it in app.py
  • Easily use Github API  from a different .py file (this file should get import to app.py) to fetch data like: https://api.github.com/users/user_name : It will fetch user name, repos, followers and many more important information.

How I used this idea for FOSSASIA (Meilix Generator)

I used Flask for the backbone of project Meilix Generator. First, I used from function to import various library needed for the project and then made several functions for the same. Let’s understand the concept using few example:

from flask import Flask, render_template
@app.route('/about')
def about():
		#About page
		return render_template("about.html")

or

from flask import Flask, send_from_directory
@app.route('/uploads/<filename>')
def uploaded_file(filename):
		return send_from_directory(app.config['UPLOAD_FOLDER'],filename)

For more details file app.py can be found here of the Meilix Generator repository where we used the above idea.

Important Links and Repositories:

Keep updating Build status in Meilix Generator

One of the problems we faced while working Meilix Generator was to provide user with the status of the custom ISO build in the Meilix Generator web app so we came up with the idea of checking the status of the link generated by the web app. If the link is available the status code would be 200 otherwise it would be 404.

We have used python script for checking the status of URL. For generating URL, we use the tag name which will be used as a variable to generate the URL of the unique event user wants the ISO for and the date will help in generation of link rest of the link remains the same.

tag = os.environ["TRAVIS_TAG"]
date = datetime.datetime.now().strftime('%Y%m%d')
url=https://github.com/xeon-zolt/meilix/releases/download/"+tag+"/meilix-zesty-"+date+"-i386.iso"

 

Now we will use urllib for monitoring the status of link.

req = Request(url)
    try:
        response = urlopen(req)
    except HTTPError as e:
        return('Building Your Iso')
    except URLError as e:
        return('We failed to reach the server.')
    else:
        return('Build Sucessful : ' + url)

 

After monitoring the status the next step was to update the status dynamically on the status page.

So we’llll use a status function in the flask app which is used by JavaScript to get status of the link after intervals of time.

Flask :

@app.route('/now')
def status_url():
    return (status())

 

Javascript:

<script type ="text/javascript">
let url ="/now"
function getstatus(url)
{
    fetch(url).then(function(response){
        return response.text()
    }).then(function(text){
        console.log("status",text)
        document.querySelector("div#status")
        .innerHTML = text
    })
    }
window.onload = function(){
    fetch(url).then(function(response){
        return response.text()
    }).then(function(text){
        console.log("status",text)
        document.querySelector("div#status")
        .innerHTML = text
    })
    window.setInterval(getstatus.bind(null,url),30*1000)
}
/*setInterval(function,interval in millsecs)*/
</script>

 

This covers various steps to prompt user whether the build is ready or not.

Resource

How Meilix Generator sends Email Notifications with SendGrid

We wanted to notify the users once the build was ready for download. To solve this we attempted making an email server on Meilix Generator but that can send email when it starts but it would take around 20 minutes to get the build ready so we thought of checking the deploy link status and send email whenever the link status was available (200) but the problem with this method was that the link can be pre available if ISO is rebuilt for same event.

Then, we attempted sending mail by Travis CI but the problem in that was closed SMTP ports (they have a strict policy about that) then we thought that Travis CI can trigger the Sendgrid which can send email to the user with the help of API.

We will use this code so that once the deployment of ISO by Travis CI is done it can execute the email script which requests Sendgrid to send email to the user.

after_deploy:
  - ./mail.py

 

We can create code using code generation service of Sendgrid we are going to choose python as it is easier to manipulate strings in python and we are going to use email as an environment variable.

After generation of python 3 code from the sendgrid website we are going to edit the message and email and hide the API key as an environment variable and create an authorization string to be used there too.

The URL will be generated by the below script as the body of url remains same only two things will change the TRAVIS_TAG which is event name and date.

date = datetime.datetime.now().strftime('%Y%m%d')
url="https://github.com/xeon-zolt/meilix/releases/download/"+os.environ["TRAVIS_TAG"]+"/meilix-zesty-"+date+"-i386.iso"

 

We can use this to hide the api key and use it as an environment variable because if the api key is visible in logs anyone can use it to exploit it and use it for spamming purpose.

authorization = "Bearer " + os.environ["mail_api_key"]
headers = {
    'authorization': authorization,

 

The main thing left to edit in the script is the message which is in the payload and is a string type so we are going to use the email received by Meilix generator as an environment variable and concatenate it with the payload string the message sent is in the value which is in the HTML format and we add the generated URL in similar way we added email variable to string.

payload = "{\"personalizations\":[{\"to\":[{\"email\":\"" + os.environ["email"] + "\"}],\"subject\":\"Your ISO is Ready\"}],\"from\":{\"email\":\"[email protected]\",\"name\":\"Meilix Generator\"},\"reply_to\":{\"email\":\"[email protected]\",\"name\":\"Meilix Generator\"},\"subject\":\"Your ISO is ready\",\"content\":[{\"type\":\"text/html\",\"value\":\"<html><p>Hi,<br>Your ISO is ready<br>URL : "+url+"<br><br>Thank You,<br>Meilix Generator Team</p></html>\"}]}"

 

The sent email looks like this

References