Deleting Meilix Github Releases

Meilix is the repository which uses build script to generate community version of lubuntu as LXQT Desktop. Meilix-Generator is the webapp which uses Meilix to generate ISO and deploy it on Meilix Github Release. Then the webapp mail the link of the ISO to the user.
Increasing number of ISO will increase the number of releases which results in dirty looking of Meilix repository. So we need to delete older releases after certain interval of time to make the repository release page looks good and decrease unwanted space.
This releases_maintainer.sh script will do this work for us.

#!/usr/bin/env bash
set -e
echo "This is a script to delete obsolete meilix iso builds by Abishek V Ashok"
echo "You have to add an authorization token to make it functional."

# jq is the JSON parser we will be using
sudo apt-get -y install jq

# Storing the response to a variable for future usage
response=`curl https://api.github.com/repos/fossasia/meilix/releases | jq '.[] | .id, .published_at'`

index=1  # when index is odd, $i contains id and when it is even $i contains published_date
delete=0 # Should we delete the release?
current_year=`date +%Y`  # Current year eg) 2001
current_month=`date +%m` # Current month eg) 2
current_day=`date +%d`   # Current date eg) 24

for i in $response; do
    if [ $((index % 2)) -eq 0 ]; then # We get the published_date of the release as $i's value here
        published_year=${i:1:4}
        published_month=${i:6:2}
        published_day=${i:9:2}

        if [ $published_year -lt $current_year ]; then
             let "delete=1"
        else
            if [ $published_month -lt $current_month ]; then
                let "delete=1"
            else
                if [ $((current_day-$published_day)) -gt 10 ]; then
                    let "delete=1"
                fi
            fi
        fi
    else # We get the id of the release as $i`s value here
        if [ $delete -eq 1 ]; then
            curl -X DELETE -H "Authorization: token $KEY" https://api.github.com/repos/fossasia/meilix/releases/$i
            let "delete=0"
        fi
    fi
    let "index+=1"
done

This code uses Github API to curl the Meilix releases. Github API is very useful in providing lots of information but here we are only concerned with the release date and time of the build.
Then we setup a condition if that satisfies then the release will automatically will get deleted.

For taking care of the authentication, a token has been uploaded to the Travis settings of Meilix of FOSSASIA.

The personal token has been generated by a user with write access to the repository with repo scope token.

This sort out the issue of having bulk of releases in the Meilix repository of FOSSASIA.

References:
Users Github API  by REST API v3
Repo Github API   by REST API v3

Wallpaper Strategy for Meilix Generator

We were first hosting the wallpaper uploaded by user on the Heroku server which was downloaded by the Travis CI which was previous solution for sending wallpaper to the Travis CI, but the problem was that wallpaper was downloaded only when build started building the ISO.

Downloading the hosted wallpaper was not a problem but the problem in that method was that the wallpaper hosted can be changed if another user also starts the build using Meilix Generator and uploads the wallpaper which will replace the previous wallpaper so simultaneous builds was not possible in previous method and resulted in conflicts.

So we thought of a sending the wallpaper to the Travis CI server for that we used base64 to and encoded it to a string using this.

with open(filename,'rb') as f:
    os.environ["Wallpaper"] = str(base64.b64encode(f.read()))[1:]

 

After uploading we send it as a variable to Travis CI and decode it. We will receive a binary file after decoding now we need to detect the mime type of the file and rename it accordingly before applying for that we use a script like this.

#renaming wallpaper according to extension png or jpg
for f in wallpaper; do
    type=$( file "$f" | grep -oP '\w+(?= image data)' )
    case $type in  
        PNG)  newext=png ;;
        JPEG) newext=jpg ;;
        *)    echo "??? what is this: $f"; continue ;;
    esac
    mv "$f" "${f%.*}.$newext"
done

 

After fixing the wallpaper extension we can apply it using the themes by replacing it with the theme wallpaper.

Resources

Base64 python documentation

How to Send a Script as Variable to the Meilix ISO with Travis and Meilix Generator

We wanted to add more features to Melix Generator web app to be able to customize Meilix ISO with more features so we thought of sending every customization we want to apply as a different variable and then use the scripts from Meilix Generator repo to generate ISO but that idea was bad as many variables are to be made and need to be maintained on both Heroku and Travis CI and keep growing with addition of features to web app.

So we thought of a better idea of creating a combined script with web app for each feature to be applied to ISO and send it as a variable to Travis CI.

Now another problem was how to send script as a variable after generating it as json do not support special characters inside the script. We tried escaping the special characters and the data was successfully sent to Travis CI and was shown in config but when setting that variable as an environment variable in Travis CI the whole value of variable was not taken as we had spaces in the script.

So to eliminate that problem we encoded the variable in the app as base64 and sent it to Travis CI and used it using following code.

To generate the variable from script.

with open('travis_script_1.sh','rb') as f:
    os.environ["TRAVIS_SCRIPT"] = str(base64.b64encode(f.read()))[1:]

 

For this we have to import base64 module and open the script generated in binary mode and using base64 we encode the script and using Travis CI API we send variable as script to the Travis CI to build the ISO with script in chroot we were also required to make changes in Meilix to be able to decode the script and then copy it into chroot during the ISO build.

sudo su <<EOF
echo "$TRAVIS_SCRIPT" > edit/meilix-generator.sh
mv browser.sh edit/browser.sh
EOF 

 

Using script inside chroot.

chmod +x meilix-generator.sh browser.sh
echo "$(<meilix-generator.sh)" #to test the file
./browser.sh
rm browser.sh

Resources

Base64 python documentation from docs.python.org

Base64 bash tutorial from scottlinux.com by Scott Miller

su in a script from unix.stackexchange.com answered by Ankit

How to customize LXQT for Meilix

We had a task of customizing the LXQT (LXQt is the Qt port and the upcoming version of LXDE, the Lightweight Desktop Environment) desktop environment in Meilix for events. For example, we can use a distro at events for presentations so during presentations things like system sounds, notifications and panel can be disturbing elements of presentations so we required LXQT to be pre configured for that so the time required in configuration is not wasted or none of the presentations are disturbed.

The default configuration of LXQT are present in ~/.config/lxqt. Configuration files are in This directory is initialized automatically. The default configuration for new users is found in /etc/xdg/lxqt but we are going to use skel for this so that if there are some changes in future in the LXQT code and they do not match default settings we can always be back to default settings by deleting user side changes. Similar to LXDE. LXQt provides a GUI applications to change its settings as well.

While Openbox is the default window manager we have used in Meilix with  LXQt, you can specify a different window manager to use with LXQt via by editing ~/.config/lxqt/session.conf. To a window manager of choice. Change the following line:

window_manager=openbox

 

We have used the Openbox for Meilix.

To create a configuration for panel we can go to ~/.config/lxqt and edit the panel.conf file if it is not present we can create a file named panel.conf and add/edit the code.

[panel1]
hidable=true 

 

GIF representing auto hide of panel.

In order to configure more things like notification we can edit the ~/.config/lxqt/notification.conf.

We can change things like the placement or the size of notification or the timeout of notification in this file.

For eg:

 [General]
__userfile__=true
placement=top-left
server_decides=1
spacing=6
width=295

 

We can now place all the configurations inside the skel folder so that every new user gets the same configurations we have made.

Resources

Shorten the Travis build time of Meilix

Meilix is a lubuntu based script. It uses Travis to compile and deploy the iso as Github Release. Normally Travis took around 17-21 minutes to build the script and then to deploy the iso on the page. It is quite good that one gets the iso in such a small interval of time but if we would able to decrease this time even more then that will be better.

Meilix script consists of basically 2 tests and 1 deploy:

The idea behind reducing the time of Meilix building is to parallely run the tests which are independent to each other. So here we run the build.sh and aptRepoUpdater.sh parallely to reduce the time upto some extent.
This pictures denotes that both the tests are running parallely and Github Releases is waiting for them to get complete successfully to deploy the iso.

Let’s see the code through which this made possible:

jobs:
  include:
    - script: ./build.sh
    - script: 'if [ "$TRAVIS_PULL_REQUEST" = "false" ]; then bash ./scripts/aptRepoUpdater.sh; fi'
    - stage: GitHub Release
      script: echo "Deploying to GitHub releases ..."

Here we included a job section in which we wrote the test which Travis has to carry out parallely. This will run both the script at the same time and can help to reduce the time.
After both the script run successfully then Github Release the iso.

Here we can see that we are only able to save around 30-40 seconds and that much matters a lot in case if we have more than 1 build going on the same time.

Links to follow:
Travis guide to build stages

Firefox Customization for Meilix

Meilix a lightweight operating system can be easily customized. This article talks about the way one must proceed to customize the configuration of Firefox of Meilix or on its own Linux distro and how to copy the configuration file of Firefox directly to the home folder of the user.
Meilix script contains a pref.js file which is responsible for providing the configuration. This file contains various function through which one can edit them according to its need to get the required configuration of its need.

Let’s see an example:

user_pref("browser.startup.homepage", "http://www.google.com/cse/home?cx=partner-pub-6065445074637525:8941524350");

This line is used to set the browser startup page and it can be edited according to user choice to find the same page whenever he starts his Firefox.

There are several lines too which can be edited to make the required changes.

How does this work?

This is the Mozilla User Preference file and should be placed in the location /home/user_name/.mozilla/firefox/*.default/prefs.js. It actually controls the attributes of Firefox preference and set the command from here to change it.

How to use it?

One can directly go and edit it according to the choice to use it.

How meilix script uses it to change the user preference?

As we can see that .mozilla folder should be under the home directory, therefore we copy the .mozilla to the the skel folder, so that it gets automatically copied to the home location and we would be able to use.
There is also a shell script which comes in handy to implement the browser startup page and the script even copies the configuration file.

1.#!/bin/bash

2.# firefox
3.# http://askubuntu.com/questions/73474/how-to-install-firefox-addon-from-command-line-in-scripts
4.for user_name in `ls /home/`
5.do
  6.preferences_file="`echo /home/$user_name/.mozilla/firefox/*.default/prefs.js`"
  7.if [ -f "$preferences_file" ]
  8.then
    9.echo "user_pref(\"browser.startup.homepage\", \"https://google.com/\");" >> $preferences_file
  10.fi
11.done

This file is taken from here. And it used in Meilix to run the script to set the default startup page in Firefox. This will be taken input from the user end from the Meilix Generator webapp and it will change the line 9 url according to the input given by the user.
On line 3, *.default will set automatically by the script itself, it generated randomly.
After that, the script will copy the prefs.js in its location and it will implement the changes.

Links to follow:

Firefox-preference guide
Firefox-editing-configuration

Generate Requirement File for Python App for Meilix-Generator

Meilix-Generator is based upon Flask (a Python framework) which has several dependencies to fulfill before actually running the app properly. This article will guide you through the way I used it to automatically generate the requirement file for Meilix Generator app so that one doesn’t have to manually type all the requirements.

An app powered by Python always has several dependencies to fulfill to run the app successfully. The app root directory contains a file named as requirements.txt which contains the name of the dependency and their version. There are features ways to generate the requirement file for an app but the one which I will demonstrate is the best one. So I used this idea to generate the requirement file for webapp Meilix Generator.

Ways to get the requirement.txt

The internet has a featured way through which one has just to run a command to get a list of all the different dependencies within an app.

pip freeze > requirements.txt

This way will generate a bunch of dependencies that we not even required.

Why do we really require to generate a requirement file?

Yes, one may even ask that we can even write the dependency in the requirements.txt file. Why do we need a command to generate it?

Since because it will take care of two important things:
1. It will ensure that all the dependencies have been included, from user input one may forget to find some of the dependency and to include that.

  1. It will also take care of the Python Package Version Pinning which is really important. People use to version pinning for Python requirements as “>=” style. It’s important to follow “==” style because If we want to install the program in one year in the future, the required packages should be pinned to assure that the API changes in the installed packages do not break the program. Please read here for more info.

The way mentioned below will ensure to provide both these features.

How I generated it for Meilix Generator?

Meilix Generator run on Flask that require a requirement.txt file to fulfill the dependencies. Let’s get straight to the way to generate it for the project.

First we will simply create a requirements.in file in which we will simply mention all the dependencies in a simple way:

Flask
gunicorn
Werkzeug

Now we will use a command to latest packages:

pip install --upgrade -r requirements.in

#Note that if you would like to change the requirements, please edit the requirements.in file and run this command to update the dependencies

Then type this command to generate the requirements.txt file from requirements.in

pip-compile --output-file requirements.txt requirements.in

#fix the versions that definitely work for an eternity.
This will generate a file something as:

click==6.7                # via flask
Flask==0.12.2
gunicorn==19.7.1
itsdangerous==0.24        # via flask
Jinja2==2.9.6             # via flask
MarkupSafe==1.0           # via jinja2
Werkzeug==0.12.2          # via flask

Now you generated a perfect requirements.txt file with all the dependencies satisfied with proper python package pinning.

The meilix-generator repo which uses this:
https://github.com/fossasia/meilix-generator

Heroku Deployment through Travis for Meilix-Generator

This article will tell the way to deploy the Meilix Generator on Heroku with the help of Travis. A successful deployment will help as a test for a good PR. Later in the article, we’ll see the one-button deployment on Heroku.

We will here deploy Meilix Generator on Heroku. The way to deploy the project on Heroku is that one should connect its Github account and deploy it on Heroku. The problem arises when one wants to deploy the project on each and every commit. This will help to test that the commits are passing or not. Here we will see that how to use Travis to deploy on Heroku on each and every commit. If the Travis test passed which means that the changes made in the commit are implemented.
We used the same idea to test the commits for Meilix Generator.

Idea behind it

Travis (.travis.yml) will be helpful to us to achieve this. We will use this deploy build to Heroku on each commit. If it gets successfully deployed then it proves that the commit made is working.

How to implement it

I will use Meilix Generator repository to tell the way to implement this. It is as simple as editing the .travis.yml file. We just have to add few lines to .travis.yml and hence it will get deployed.

deploy:
  provider: heroku
  api_key:
    secure: "YOUR ENCRYPTED API KEY"     # explained below
  app: meilix-generator                    # write the name of the app
  on:
    repo: fossasia/meilix-generator                # repo name
    branch: master                        # branch name

Way to generate the api key:
This is really a matter of concern since if this gets wrong then the deployment will not occur.

Steps:

  • cd into the repository which you want to deploy on Heroku.
  • Login Heroku CI and Travis CI into your terminal and type the following.

travis encrypt $(heroku auth:token) --add deploy.api_key

This will automatically provide the key inside the .travis.yml file.

You can also configure manually using

travis heroku setup

That it, you are done, test the build.

Things are still left:

But we are still left with the test of the PR.
For this we have to create a new app.json file as:

{
    "name": "Meilix-Generator",
    "description": "A webapp which generates iso for you",
    "repository": "https://github.com/fossasia/meilix-generator/",
    "logo": "https://github.com/fossasia/meilix-generator/blob/master/static/logo.png",
    "keywords": [
        "meilix-generator",
        "fossasia",
        "flask"
    ],
    "env": {
        "APP_SECRET_TOKEN": {
            "generator": "secret"
        },
        "ON_HEROKU": "true",
        "FORCE_SSL": "true",
        "INVITATION_CODE": {
            "generator": "secret"
        }
    },
    "buildpacks": {
            "url": "heroku/python"        # this is the only place of concern
        }
}

This code should be put in a file in the root of the repo with the name as app.json.
In the buildpacks : the url should be the one which contains the code base language used.

This can be helpful in 2 ways:

  1. Test the commit made and deploy it on Heroku
    2. One-click deployment button which will deploy the app on Heroku
  • Test the deployment through the URL:

https://heroku.com/deploy?template=https://github.com/user_name/repo_name/tree/master

  • Way to add the button:

[![Deploy](https://www.herokucdn.com/deploy/button.svg)](https://heroku.com/deploy)

How can this idea be helpful to a developer

A developer can use this to deploy its app on Heroku and test the commit automatically and view the quality and status of PR too.

Useful repositories and link which uses this:

I have used the same idea in my project. Do have a look:

https://github.com/fossasia/meilix-generator
deployment on Heroku
one-click deployment
app.json file schema   

Customizing Firefox in Meilix using skel

We had a problem of customizing the Firefox browser configuration using Meilix generator so we used the skel folder in Linux. The /etc/skel directory contains files and directories that are automatically copied over to a new user’s home directory when such user is created by the useradd program.

Firefox generally reads its settings from ~/.mozilla/firefox folder present in Linux home of the user, which is changed when user modifies his settings like changing the homepage or disabling the bookmarks. To add these changes for every new user we can copy these configurations into skel folder.

Which we have done with Meilix which have these configurations in it and can be modified by using Meilix generator.

If you are going to look in your ~/.mozilla/firefox folder you will find a random string followed by .default folder this will contain the default settings of your Firefox browser and this random string is created uniquely for every user. The profile name will be different for all users but should always end with .default so if we use skel the .default will be same for everyone and we may choose any name for that like we have used meilix.default in below script.

For example to change the default homepage URL we can create a script that can be used by Meilix generator to modify the URL in Meilix.

#!/bin/bash

set -eu   	 
# firefox
preferences_file="`echo meilix-default-settings/etc/skel/.mozilla/firefox/meilix.default/user.js`"
if [ -f "$preferences_file" ]
then
	echo "user_pref(\"browser.startup.homepage\", \"${event_url}\");" >> $preferences_file
fi

 

Here the ${event_url} is the environment variable sent by Meilix generator to be added to configuration of Firefox that is added to skel folder in Meilix.

We can similarly edit more configuration by editing the file pref.js file it contains the configuration for Firefox like bookmarks, download location, default URL etc.

Resources

Setting Environment Variables up in Travis and Heroku for Meilix and Meilix-Generator

Meilix Generator is a webapp whose task is to take input as a configuration and start the Meilix build. But an anonymous person cannot start the Meilix build of any user and deploy the release in the repository. There are ways which are used as authentication passes through environment variable to start the build. In this article, I show the way I used to trigger Meilix by setting up environment variables in Meilix Generator.

Environment variables are of great use when one has to supply personal token in an open-source project for accessing the repository. So down there, we will have ways to configure the variables in Heroku and Travis. There are so many wikis out there but this one is the blend of both Heroku and Travis.

Heroku

There are several ways to setup variables in Heroku. The way I’m going to describe below is used to access Travis build using Heroku.
Using the Heroku variable generated using Travis will help to trigger the build on Travis.

How:

Idea:
We will use Travis CLI to generate a token (unique and keep it secret). Then provide the token as a variable name to the Heroku.
Backdoor:
This Travis token will give access to the Heroku to trigger the build on that particular Travis account. We use variable to provide the token since in the script we will use this variable as an environment variable to fetch the token in the place of token like as $token.
Implementation:
Open your terminal and type the following:

sudo apt install ruby ruby-dev
sudo gem install travis                       	# install Travis CLI
travis login --org   					# login into Travis
travis token --org		# generate your secret personal access token

You will get a token, copy and paste it into your Heroku app’s settings config vars token. You have to use the `KEY` as the variable which is used in the script for triggering the build. Save it and you are done the setting of the token in the Heroku.

Travis

Now it’s time for Travis token.
It is used to deploy the build to that repository only.
We can use the token in two ways either paste it in the setting of that repository on Travis or pasting the encrypted form of that in the .travis.yml file in that repository. Both will work. But one thing to remember that you must have the write access to that repository.

How

,Idea:
It is used in .travis.yml file as an environment variable to successfully build and deploy the application as a Github release.
Backdoor:
The token gives the permission to Travis to deploy the build application in the GitHub release of the repo(if one using to deploy it there only).
Implementation:
Head up to Github and generate a personal access token with scope repo. Copy the generated token in a safe place.

Way 1:

Paste the token in the setting of the repo in Travis in Environment Variable option. Now it will access the Github repository since it has got the permission from the personal token generated from Github.

Way 2:
Open terminal:

cd repo_name				# cd into the cloned repo
travis encrypt secret_token	#replace secret_token with the token generated

or

travis encrypt secret_token -r user/repo 	#if you are not in the repo

Copy that encrypted token and paste it in proper format in the .travis.yml file. Now you enabled Travis giving permission to release the build.

How can this idea be helpful to a developer

A developer can use this to build the Github Release in its repository. One can secure its token using this technique. One can use it to trigger its personal project in Travis using Heroku.

Useful repositories and link which uses this:

I have used the same idea in my project. Do have a look https://github.com/fossasia/meilix-generator
about environment variable
encryption keys
triggering build