Speeding up the Travis Build to Decrease the Building Time

Meilix is the repository which uses build script to generate community version of lubuntu as LXQT Desktop. It usually takes around 25-26 to build and deploy the ISO as a Github Release on master branch.
Observing the build log we can see that there are most of the packages like debootstrap, squashfs-tool, etc which are being fetch and setup at the time of building which results in increasing build time.

The issue is to decrease the build time supplying the packages from Travis so that when we run build.sh we won’t be required to download them again and thus few minutes will get reduced.
We included list of packages to be pre-downloaded in .travis.yml

include:
  - os: linux
    addons:
      apt:
        sources:
          - ubuntu-toolchain-r-test
        packages:
          - debootstrap
          - genisoimage
          - p7zip-full
          - squashfs-tools
          - ubuntu-dev-tools
          - dpkg-dev
          - debhelper
          - fakeroot
          - devscripts

These are some important packages included in the build.sh  as devtools=”debootstrap genisoimage p7zip-full squashfs-tools ubuntu-dev-tools” which are always fetched, so we included it into .travis.yml and downloaded it before entering into the chroot environment. By specifying those packages in the .travis.yml, Travis provides those packages beforehand into the docker container so it will run our script. Since the scripts also include package then when the script runs apt-get it won’t download those packages again. They are specified outside the chroot environment because they are expected to be at the system the build.sh script is run to get the iso. By this way, we get a sharp decrease in build time as the internet in the Travis CI container is not so fast so the package download time can be avoided. Now the build is taking around 15-16 minutes to build and deploy.

One thing to note that we didn’t remove those packages off build.sh so that build.sh works outside Travis CI as well.

References:
Pull #176 by @abishekvashok
Speeding up Travis Build by Travis CI
Faster Build by atchai.com

Functionality and Customization of the Meilix Metapackage meilix-default-settings

Meilix has is made of build file and metapackages. Build file is responsible for executing commands and successfully implementing the work of metapackages.

Metapackages in Meilix
Name of metapackages used in Meilix are: meilix-artwork, meilix-default-settings.

meilix-default-settings

meilix-default-settings have 3 major folders debian, etc and usr and a Makefile. We are only concerned with etc and usr folder here.
etc and usr folders are folders in which if changes are made that can be seen the ISO. One can assume this as two folders present in the root folder of a Linux Distro.

Its directory is like this:

meilix-artwork

meilix-artwork has 1 main folder named as usr which contain share folder in which plymouth configuration is made. One can make changes here and it will directly seen in the Linux Distro.

Its directory looks like this:

How these meta packages actually work?
To get the answer one has to jump into the debian folder of any of the metapackage. It contains a control file. This contains information of the metapackages.

Source: meilix-default-settings
Section: x11
Priority: extra
Maintainer: meilix <vanhonit@gmail.com>
Build-Depends: debhelper (>= 8.0.0)
Standards-Version: 3.9.2
Homepage: http://mbm.vn

Package: meilix-default-settings
Architecture: all
Depends: ${shlibs:Depends}, ${misc:Depends}
Description: default settings for meilix
 Various system settings tailored for meilix.

One can update the metapackage from here and tweak with its depends. One come to know about the maintainer of the metapackage which can contacted in case of any issue. We can also know for which architecture this metapackage is made and about its description.
The whole debian does the work but after making any changes in the metapackage, it needs to be rebuild which is performed by debuild.sh. This is how a metapackages in Meilix works.

References:
Linux MetapackagesMatthartley from linux.com
Creating a MetapackageAjmitch from askubuntu.com

Deleting Meilix Github Releases

Meilix is the repository which uses build script to generate community version of lubuntu as LXQT Desktop. Meilix-Generator is the webapp which uses Meilix to generate ISO and deploy it on Meilix Github Release. Then the webapp mail the link of the ISO to the user.
Increasing number of ISO will increase the number of releases which results in dirty looking of Meilix repository. So we need to delete older releases after certain interval of time to make the repository release page looks good and decrease unwanted space.
This releases_maintainer.sh script will do this work for us.

#!/usr/bin/env bash
set -e
echo "This is a script to delete obsolete meilix iso builds by Abishek V Ashok"
echo "You have to add an authorization token to make it functional."

# jq is the JSON parser we will be using
sudo apt-get -y install jq

# Storing the response to a variable for future usage
response=`curl https://api.github.com/repos/fossasia/meilix/releases | jq '.[] | .id, .published_at'`

index=1  # when index is odd, $i contains id and when it is even $i contains published_date
delete=0 # Should we delete the release?
current_year=`date +%Y`  # Current year eg) 2001
current_month=`date +%m` # Current month eg) 2
current_day=`date +%d`   # Current date eg) 24

for i in $response; do
    if [ $((index % 2)) -eq 0 ]; then # We get the published_date of the release as $i's value here
        published_year=${i:1:4}
        published_month=${i:6:2}
        published_day=${i:9:2}

        if [ $published_year -lt $current_year ]; then
             let "delete=1"
        else
            if [ $published_month -lt $current_month ]; then
                let "delete=1"
            else
                if [ $((current_day-$published_day)) -gt 10 ]; then
                    let "delete=1"
                fi
            fi
        fi
    else # We get the id of the release as $i`s value here
        if [ $delete -eq 1 ]; then
            curl -X DELETE -H "Authorization: token $KEY" https://api.github.com/repos/fossasia/meilix/releases/$i
            let "delete=0"
        fi
    fi
    let "index+=1"
done

This code uses Github API to curl the Meilix releases. Github API is very useful in providing lots of information but here we are only concerned with the release date and time of the build.
Then we setup a condition if that satisfies then the release will automatically will get deleted.

For taking care of the authentication, a token has been uploaded to the Travis settings of Meilix of FOSSASIA.

The personal token has been generated by a user with write access to the repository with repo scope token.

This sort out the issue of having bulk of releases in the Meilix repository of FOSSASIA.

References:
Users Github API  by REST API v3
Repo Github API   by REST API v3

Updating of Linux Standard Base (lsb) and Shorten of log of Meilix Build

Updating the Linux Standard Base of the Meilix

Originally Meilix uses the Ubuntu mirror to fetch the Kernel source for the building of the Operating System. Therefore whenever a user type lsb_release -a for fetching the required information, they get the Ubuntu build info. The task is to change the config file to update the Linux Base Information from Ubuntu to Meilix.

lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 17.04
Release: 17.04
Codename: zesty

We need to patch a file in the location meilix-default-settings/etc/lsb-release which contains the information of the lsb release and this will overwrite the original configuration of Meilix.
This is how the lsb-release file looks like now:

DISTRIB_ID=Meilix
DISTRIB_RELEASE=17.04
DISTRIB_CODENAME=meilix
DISTRIB_DESCRIPTION="Meilix 17.04"

We made the required changes in the file as shown above and the output is as follows:

Shorten the log length of Meilix in Travis

We are facing issues while deployment of Meilix ISO in Travis and the error follows is ReadTimeout Error. Example log of one of fail build is:

This error gets solved automatically and the the ISO gets deployment after 1 or 2 restart of the build. But this time the error doesn’t get solved and after several attempts of restart of build, the ISO doesn’t get deployed.

Reason behind the error:

Travis is taking a lot of time to build the ISO. Travis logs are exceeding the time limit.

Proposed solution:

Reduce the time of build or shift to a new CI.
Reduce the log of the build so as to get the log within 9999 lines.

Solution Implemented

The best solution is to reduce the number of lines used in the log and this will also reduce the time of the build.
I tried concealing some command outputs by appending >/dev/null 2>&1 to some of the commands that has long outputs and adding -y to the commands like:

apt-get -qq -y --purge install file-roller unrar

References

Wiki of Linux Standard Base
Linux Foundation lsb
Ubuntu answer to reduce log

Turning off Power Management and Blanking of Screen in Meilix

Meililx has the LXQt desktop environment which already have the lxqt-powermanagement package. We don’t need to configure the power management during an event. Every time when an ISO is booted a pop-up arrives which asks to configure power. On the one hand, the pop-ups are a disturbing element during a presentation, slide-show, etc. On the other hand, while a presentation is going on and the presenter is explaining a slide for a longer duration then there is a possibility of the screen to turn blank or screen saver to get started due to inactivity. In this post we will discuss to resolve the above two issues.

Disabling Power Management
The issue has been filed #133 to solve. We can follow two approaches to shutdown the power daemon which is responsible for starting the power-management everytime during startup.

In the beginning the issue looks as:

One is to copy the script to /etc/X11/xorg.conf.d/10-monitor.conf by editing the meilix-default-settings metapackage. 

1. [panel1]
2. alignment=-1
3. animation-duration=0
4. background-color=@Variant(\0\0\0\x43\0\xff\xff\0\0\0\0\0\0\0\0)
5. background-image=
6. desktop=0
7. font-color=@Variant(\0\0\0\x43\0\xff\xff\0\0\0\0\0\0\0\0)
8. hidable=true
9. iconSize=22
10. lineCount=1
11. lockPanel=false
12. opacity=100
13.panelSize=32
14. plugins=mainmenu, desktopswitch, quicklaunch, taskbar, tray, statusnotifier, mount, volume, clock, showdesktop
15. position=Bottom
16. show-delay=0
17. width=100
18. width-percent=true

Explanation of code:

In the Monitor section, DPMS (Desktop Power Management System) is turned off by setting the value to false
In the ServerLayout section, Blanktime, Offtime, SuspendTime, StandbyTime are set to “0”, so that they will be disabled.
Through this way we are suspending the power-management to use it feature by making changes in the script. This approach will make sure that this feature can’t be changed through the GUI, too.

And the other approach is the basic one but it’s also fulfilling the similar need. Adding a line to chroot.sh

apt-get purge lxqt-powermanagement

Both will solve the issue and the pop will not arrive at the startup of the OS.
Now LXQt desktop starts as:

Disabling Blanking of Screen in Meilix
A presenter never wants to turn his monitor blank or screensaver while presenting his presentation. Here we will edit the .profile file in the home folder to fulfill our requirement.
For get the same file in the ISO, we need to patch the file meilix-default-settings/etc/skel/.profile.
Skel folder has the property to transfer the tree inside the folder to the home folder of the new user created. For example here, the new user created will have .profile folder inside its home directory and it will apply the requested changes.

Important code which is responsible for turning the blanking of screen off:

xset s off && xset -dpms

 

Link to important pages:
Disabling Power Management – Arch Wiki
Code for turning Power Management (PR) – @meets2tarun (author)
Code for turning off blanking of screen (PR) – @meets2tarun (author)

Configuration for Auto-hiding Panel in Meilix with LXQt desktop

In the new LXQt desktop, we can intelligently hide the panel. For that purpose, we’ll just need to patch a new file in a location under the meilix-default-settings metapackage.
Originally the file lies in the lxqt folder of the .config of the OS with the name panel.conf like .config/lxqt/panel.conf but since we have to make changes in the metapackage, we need to patch it here meilix-default-settings/etc/skel/.config/lxqt/panel.conf. Files in etc/skel/ will be put in each users’ new home folder, a folder which does not exist yet when we build the ISO.

panel.conf

1. [panel1]
2. alignment=-1
3. animation-duration=0
4. background-color=@Variant(\0\0\0\x43\0\xff\xff\0\0\0\0\0\0\0\0)
5. background-image=
6. desktop=0
7. font-color=@Variant(\0\0\0\x43\0\xff\xff\0\0\0\0\0\0\0\0)
8. hidable=true
9. iconSize=22
10. lineCount=1
11. lockPanel=false
12. opacity=100
13.panelSize=32
14. plugins=mainmenu, desktopswitch, quicklaunch, taskbar, tray, statusnotifier, mount, volume, clock, showdesktop
15. position=Bottom
16. show-delay=0
17. width=100
18. width-percent=true

In the line number 8 , hidable=true is doing all the jobs. It is the only line which hides the panel by default.

How we find this approach?
Originally LXQt panel is not hidden, they are shown by default. I first try to locate panel.conf file which will carry out the configuration for the panel. I try to find the code responsible for hiding the panel, but I can’t find that. Then I copied the panel.conf in a file and then by GUI I hide the panel and reopen the config file. Then I compare the changes between this file and the old config.panel file in which I found that the new file has a new line hidable=true. We introduced the changes in this PR.

How this approach actually work?
We are using meilix-default-settings metapackage to make the things work. We made an .config file which contains the configuration file. And the .config file is present under skel folder which gets copied under the home folder of the user. Thus ultimately we get a configuration file which will overwrite the original one to get the desired changes.

Other Uses of panel.conf
The file panel.conf could be used to customize all related settings to the LXQT panel, like its alignment, volume bar, quick launch, show desktop, etc.

References:
LXQt panel hiding
Customize LXQt desktop

Debuilding the meilix-default-settings Metapackage

In the Meilix code repository you find a metapackage named meilix-default-settings which contains custom settings in directories as debian, etc, and user. In these directories one can make changes to make them be included in the build ISO. As Meilix runs on Debian we package our custom user settings in a Debian package to be installed along all the other software packages. The process and utility to make a Debian package is called debuild.

Directories in the meilix-default-settings:

What is debuilding?

It’s Debian slang for “making a deb package” and that stirred quite some confusion in our communications. Debuild is actually a rebuilding of the metapackage. But as to rebuild the Debian package you usually type debuild -uc -us therefore I stick to the language

Suppose someone has edited a configuration file in the metapackage according to its desires to achieve a specific result in the ISO it won’t get in unless he rebuilds the metapackage.He has not only to edit the metapackage but also to rebuild it to get the desired output in the ISO. To make the process automated, we have made a tiny script which will debuild the metapackages during each and every build, we only need to modify the metapackage.

Actually the first meilix-default-settings folder is the only metapackage and inside of it is the sub-metapackage which is responsible to get the changes applied in the ISO. To see a change in the ISO, we only need to edit the meilix-default-settings usr or etc folder in the first layer. Then, we need to debuild the metapackages.

Code-Base:

This file is present here

1. #!/bin/bash
2. rm meilix-default-settings_*                                    
3. cd meilix-default-settings                                      
4. debuild -uc -us

Let’s go through the whole code base line by line:
Line 2 deletes the previous meilix-default-settings binary packages.
Line 3 in this we changed our directory to the metapackage folder that is of our concern.
Line 4 is the most important line, it builds the whole metapackage and brings back all the binary packages and metapackages after making the desired changes.

Follow the example below to know that actually how it works:

This pull request is responsible to turn off system sounds by default in the generated ISO. Pull Requests files in which I only edited the this file and rest of the files get changes in the process of debuilding the metapackage (ignore .travis.yml file).

References:
Required files under debian directory
Debian directory guideline

Migration of LXDE Desktop of Meilix to LXQt

Meilix is originally based on LXDE and our task is to migrate Meilix desktop to LXQt. Originally LXQt is a fusion of LXDE and the Razor-Qt desktop.

What is LXDE and LXQt ?
Both are desktop environment. They are light weight with a beautiful GUI and a user-friendly touch.

Older code to install a LXDE desktop in a Debian-based environment is (source):

apt-get -q -y --purge install --no-install-recommends \
lxsession lightdm lightdm-gtk-greeter lxterminal gvfs-backends seahorse \
network-manager-gnome xorg dbus-x11 openbox ubiquity-frontend-gtk vlc \
xfce4-mixer gstreamer0.10-alsa pulseaudio pavucontrol lxpanel \
mozilla-plugin-vlc lubuntu-core jockey-gtk

In the chroot environment, the maintainer has pick up the it’s dependencies and recommend packages and listed all packages which required to install a LXDE desktop. He keeps the size as small as possible. The maintainer had hand-picked only the required packages which can run the desktop.

What we did:
We tried with several approaches. Few worked and few did not. We removed all the packages related to desktop like gnome-games, gdm and gnome-language-en since we don’t want any sort of problem conflicts in the new desktop. We had also remove all the lines mentioned above which install LXDE.

Then I simply typed the line:

apt-get -q -y install lxqt

This way we only reach to the CLI version of the OS, and we actually don’t know whether we actually have the desktop install or not.
We tried to install lighdm their by

sudo apt-get install xinit

but that was also giving errors.

We changes the line to:

apt-get -q -y install xorg sddm lxqt

And suddenly the same approach which brought to a black screen and that was a desktop and the window manager was openbox.

Then we tried to add sddm.conf to get the desktop and realized that sddm starts plasma instead of LXQt. Then we added a config file for sddm to start up LXQt by default. This file is present in the location `/etc/sddm.conf` in the meilix-default-settings metapackage with the code as:

[Autologin]
User=hotelos
Session=lxqt.desktop

But due to a bug reported here, it still starts plasma instead of LXQt. So now I have to patch a script in the location /usr/share/initramfs-tools/scripts/casper-bottom/15autologin

In last paragraph of the script it will first detect the presence of sddm, if that exists, it will assume that plasma will be default desktop and try to detect Lubuntu.desktop and QLubuntu.desktop .
So, change plasma.desktop to lxqt.desktop.

In the 15autologin changes line 84:

if [ -f /root/usr/bin/sddm ]; then
	sddm_session=lxqt.desktop
	if [ -f /root/usr/share/xsessions/Lubuntu.desktop ]; then
    	    sddm_session=Lubuntu.desktop
	fi
	if [ -f /root/usr/share/xsessions/QLubuntu.desktop ]; then
    	    sddm_session=QLubuntu.desktop
	fi
fi

Here, sddm is made to configure it must use lxqt desktop.

And this worked. Finally we get the LXQt desktop for Meilix.

Since this script along with other scripts will be packed into a small filesystem called initramfs, therefore we have to write `update-initramfs -u ` after the installation of the meilix-default-settings package.

I had made a table of the things that worked and that didn’t.

Things which do not work Things which work
1. Firstly, we removed all these lines and replace it with `apt-get -q -y install lxqt`, but this will take us to a CLI and ask to install xinit which takes us to another error. We want to straight forward go to GUI.
2. Removing of all the others desktop dependencies like gnome (here), LXDE. The way build.sh was written is that it includes suggests and recommends packages to take less space.
3. Giving less RAM to the live boot ISO which gives the error in the installation of xinit in CLI mode.
4. Removing single line and adding single line and keep on testing rather than testing on a big code. This helped us to find that where actually the issue lies.
5. Removing the rc.xml file which is responsible for openbox configuration that we don’t need in the LXQt. Screenshots are attached below.
6. Finally I booted into CLI and there I type startx which ask me to do apt-get install xinit then after that I did startx from where it takes me to the black screen.
7. I included lightdm but it’s not working and when i tried to start it using the terminal of openbox “systemctl start lightdm” , it gives the error as “failed to start lightdm.service: Unit lightdm.service not found”.
8. When I tried to install lightdm “sudo apt-get install lightdm-gtx-greeter” and we get the error pasted below.
9. Then I though to go for sddm which is a “Simple Desktop Display Manager” a replacement for lightdm.
10. Finally we make it something like this “sudo apt-get install -q -y xorg sddm lxqt”
11. sddm starts plasma (KDE) instead of LXQt for getting the desire result we need is to add /etc/sddm.conf and set its login default to lxqt.desktop
12. After screenshot of number 10, I clicked ok and logout myself to CLI and login and type startx to get the screenshot.
13. All of the things that I was doing is in the xenial version of meilix and when I tried the same in zesty. I get the following result.
14. We think there must be something why Casper uses plasma by default instead of lxqt. We can also try preventing plasma.desktop being installed in the first place.
15. Casper will set up the login when live cd is started, and the file will be changed too.
So writing to the config file (sddm.conf) actually does nothing.
16. apt install lubuntu installs lubuntu.desktop while Casper (It is actually responsible for setting up live environment, which also needs to takes care of auto login) checks for Lubuntu.desktop (noting the capital letter)
It is in the script `/usr/share/initramfs-tools/scripts/casper-bottom/15autologin`
Probably we have to patch that.
So Casper will write configs to login manager like sddm.At the time of booting the ISO into the live desktop.
The problem is, the script Casper is using, detects a different file name for LXQt.
17. We get the “15autologin file” inside chroot directory if you build the iso by yourself.
18. Then we added the line to update – initramfs (initramfs is a small filesystem that linux will mount first, which will only have basic functions, then other partitions).

Number 3

xinit error after increasing of the RAM

Number 5

Number 6

This is Openbox.

Number 8

Number 10

Number 12

Number 13.

Finally the desktop LXQt

References:
LXQt Desktop
LXDE Desktop
Switching Desktop Environment

Shorten the Travis build time of Meilix

Meilix is a lubuntu based script. It uses Travis to compile and deploy the iso as Github Release. Normally Travis took around 17-21 minutes to build the script and then to deploy the iso on the page. It is quite good that one gets the iso in such a small interval of time but if we would able to decrease this time even more then that will be better.

Meilix script consists of basically 2 tests and 1 deploy:

The idea behind reducing the time of Meilix building is to parallely run the tests which are independent to each other. So here we run the build.sh and aptRepoUpdater.sh parallely to reduce the time upto some extent.
This pictures denotes that both the tests are running parallely and Github Releases is waiting for them to get complete successfully to deploy the iso.

Let’s see the code through which this made possible:

jobs:
  include:
    - script: ./build.sh
    - script: 'if [ "$TRAVIS_PULL_REQUEST" = "false" ]; then bash ./scripts/aptRepoUpdater.sh; fi'
    - stage: GitHub Release
      script: echo "Deploying to GitHub releases ..."

Here we included a job section in which we wrote the test which Travis has to carry out parallely. This will run both the script at the same time and can help to reduce the time.
After both the script run successfully then Github Release the iso.

Here we can see that we are only able to save around 30-40 seconds and that much matters a lot in case if we have more than 1 build going on the same time.

Links to follow:
Travis guide to build stages

Firefox Customization for Meilix

Meilix a lightweight operating system can be easily customized. This article talks about the way one must proceed to customize the configuration of Firefox of Meilix or on its own Linux distro and how to copy the configuration file of Firefox directly to the home folder of the user.
Meilix script contains a pref.js file which is responsible for providing the configuration. This file contains various function through which one can edit them according to its need to get the required configuration of its need.

Let’s see an example:

user_pref("browser.startup.homepage", "http://www.google.com/cse/home?cx=partner-pub-6065445074637525:8941524350");

This line is used to set the browser startup page and it can be edited according to user choice to find the same page whenever he starts his Firefox.

There are several lines too which can be edited to make the required changes.

How does this work?

This is the Mozilla User Preference file and should be placed in the location /home/user_name/.mozilla/firefox/*.default/prefs.js. It actually controls the attributes of Firefox preference and set the command from here to change it.

How to use it?

One can directly go and edit it according to the choice to use it.

How meilix script uses it to change the user preference?

As we can see that .mozilla folder should be under the home directory, therefore we copy the .mozilla to the the skel folder, so that it gets automatically copied to the home location and we would be able to use.
There is also a shell script which comes in handy to implement the browser startup page and the script even copies the configuration file.

1.#!/bin/bash

2.# firefox
3.# http://askubuntu.com/questions/73474/how-to-install-firefox-addon-from-command-line-in-scripts
4.for user_name in `ls /home/`
5.do
  6.preferences_file="`echo /home/$user_name/.mozilla/firefox/*.default/prefs.js`"
  7.if [ -f "$preferences_file" ]
  8.then
    9.echo "user_pref(\"browser.startup.homepage\", \"https://google.com/\");" >> $preferences_file
  10.fi
11.done

This file is taken from here. And it used in Meilix to run the script to set the default startup page in Firefox. This will be taken input from the user end from the Meilix Generator webapp and it will change the line 9 url according to the input given by the user.
On line 3, *.default will set automatically by the script itself, it generated randomly.
After that, the script will copy the prefs.js in its location and it will implement the changes.

Links to follow:

Firefox-preference guide
Firefox-editing-configuration