Speeding up the Travis Build to Decrease the Building Time

Meilix is the repository which uses build script to generate community version of lubuntu as LXQT Desktop. It usually takes around 25-26 to build and deploy the ISO as a Github Release on master branch.
Observing the build log we can see that there are most of the packages like debootstrap, squashfs-tool, etc which are being fetch and setup at the time of building which results in increasing build time.

The issue is to decrease the build time supplying the packages from Travis so that when we run build.sh we won’t be required to download them again and thus few minutes will get reduced.
We included list of packages to be pre-downloaded in .travis.yml

include:
  - os: linux
    addons:
      apt:
        sources:
          - ubuntu-toolchain-r-test
        packages:
          - debootstrap
          - genisoimage
          - p7zip-full
          - squashfs-tools
          - ubuntu-dev-tools
          - dpkg-dev
          - debhelper
          - fakeroot
          - devscripts

These are some important packages included in the build.sh  as devtools=”debootstrap genisoimage p7zip-full squashfs-tools ubuntu-dev-tools” which are always fetched, so we included it into .travis.yml and downloaded it before entering into the chroot environment. By specifying those packages in the .travis.yml, Travis provides those packages beforehand into the docker container so it will run our script. Since the scripts also include package then when the script runs apt-get it won’t download those packages again. They are specified outside the chroot environment because they are expected to be at the system the build.sh script is run to get the iso. By this way, we get a sharp decrease in build time as the internet in the Travis CI container is not so fast so the package download time can be avoided. Now the build is taking around 15-16 minutes to build and deploy.

One thing to note that we didn’t remove those packages off build.sh so that build.sh works outside Travis CI as well.

References:
Pull #176 by @abishekvashok
Speeding up Travis Build by Travis CI
Faster Build by atchai.com

Creating Modified Initrd for Meilix

We are modifying the Initrd file in order to modify the live user configuration which is not available via skel but can be modified by editing the casper configuration like hostname or the systemd configuration.

Linux has the Initrd or “Initial ram-disk” used during the boot process. A Linux kernel is modular. The Kernel files, drivers, reside in separate files, i.e., kernel modules. The kernel does not require drivers as BIOS handles all the work of loading Initrd into the memory. After the kernel is loaded, it starts the boot process. Initrd contains all the drivers Linux needs to boot and the user can rebuild Initrd without changing the kernel.

Files inside Initrd

The files inside Initrd include the conf where the casper configuration is found which is required for the live user creation and its configuration.

The Initrd file is used when booting a live Meilix and can be found in the casper directory of the ISO.

We start with downloading the ISO and mounting it:

sudo mount -o loop meilix-i386.iso /mnt

Modifying the Initrd

Now extract the content into a folder so that one can modify them. It depends on the image type in the Meilix. We have the image in lz format it can be in gzip format for other distributions.

mkdir initrd-tmp
  cd initrd-tmp
lzma -dc -S .lz /mnt/casper/initrd.lz | cpio -id

 

Now we can modify the files like hostname or the files required to run during boot. The Initrd file is responsible for all the configuration available to the live user like the theme for Plymouth. We can modify the boot menu or plymouth or the files and folders the user gets like the default wallpaper.

Repack the modified files into a new initrd:

find . | cpio --quiet --dereference -o -H newc | gzip -9 > ~/new-initrd.gz

 

These are steps to modify the Initrd file which can be used to customize the Meilix live user configuration like hostname, Plymouth theme and user files.

Resources

Modifying Notifications in Meilix

There are many settings available for notifications in Meilix like position, size, timeout which can be modified with help of the notifications settings available in LXQT.

Theming Notifications

We will start by creating a file in /usr/share/lxqt/themes for creating a qss file of the notification theme as lxqt-notificationd.qss. This file tells the LXQT about what the colors, size, border, etc. are for a notification.

We start with notifications in this file. We can define color of the text of notifications. It supports alpha values too, but making text transparent decrease.

Notification {
    color: #313639;
    border: 1px solid rgba(155, 155, 155, 00%);
    background:rgba(240, 240, 240, 00%);
    margin: 0px;
    border-radius: 5px;
}

 

We can add a custom close button also we just need to add the path of the button in the qss file.

#closeButton {
    margin: 3px;
    border-radius: 4px;
    border: 1px solid transparent;
    padding: 4px;
    qproperty-icon: url(lxqt-notificationd/window-close.svg);
}

 

Other properties like hover on close button for animations can be used like.

#closeButton:hover {
    color: rgba(54, 54, 54, 100%);
    background: rgba(30, 145, 255, 30%);
    border: 1px solid rgba(30, 145, 255, 100%);
}

 

We can also define custom font, background like an image file, or actions like click, hover etc.

For other notification settings we can create a configuration file notifications.config which we can place in root  or in ~/.config depending upon application weather we want to apply it system wide or only for user.

The configuration file for disabling the sounds in LXQT:

[Sounds]
No sound=true

 

Settings can also be changed using the menu and they are saved in .config/lxqt/notifications.conf. We have changed the spacing to zero in Meilix so that the notifications are invisible and do not disturb during an event (instead of disabling it in case someone wants notifications they can change the spacing).

[General]
spacing=0

Resources

Functionality and Customization of the Meilix Metapackage meilix-default-settings

Meilix has is made of build file and metapackages. Build file is responsible for executing commands and successfully implementing the work of metapackages.

Metapackages in Meilix
Name of metapackages used in Meilix are: meilix-artwork, meilix-default-settings.

meilix-default-settings

meilix-default-settings have 3 major folders debian, etc and usr and a Makefile. We are only concerned with etc and usr folder here.
etc and usr folders are folders in which if changes are made that can be seen the ISO. One can assume this as two folders present in the root folder of a Linux Distro.

Its directory is like this:

meilix-artwork

meilix-artwork has 1 main folder named as usr which contain share folder in which plymouth configuration is made. One can make changes here and it will directly seen in the Linux Distro.

Its directory looks like this:

How these meta packages actually work?
To get the answer one has to jump into the debian folder of any of the metapackage. It contains a control file. This contains information of the metapackages.

Source: meilix-default-settings
Section: x11
Priority: extra
Maintainer: meilix <vanhonit@gmail.com>
Build-Depends: debhelper (>= 8.0.0)
Standards-Version: 3.9.2
Homepage: http://mbm.vn

Package: meilix-default-settings
Architecture: all
Depends: ${shlibs:Depends}, ${misc:Depends}
Description: default settings for meilix
 Various system settings tailored for meilix.

One can update the metapackage from here and tweak with its depends. One come to know about the maintainer of the metapackage which can contacted in case of any issue. We can also know for which architecture this metapackage is made and about its description.
The whole debian does the work but after making any changes in the metapackage, it needs to be rebuild which is performed by debuild.sh. This is how a metapackages in Meilix works.

References:
Linux MetapackagesMatthartley from linux.com
Creating a MetapackageAjmitch from askubuntu.com

LXQT Panel theme for Meilix

Panel theming is available via the LXQt Configuration Center –> LXQt Appearance –> LXQt Theme. For Meilix we have attempted making the light color theme with some new things like taskbar manager to highlight the active window and hover actions.

Meilix taskbar in panel

Theme folders of LXQT is in the directory /usr/share/lxqt/themes/.

Meilix LXQT theme directory

For theming the panel we start by creating a lxqt-panel.qss this file is like a style sheet for LXQt panel.

LXQtPanel #BackgroundWidget {
        background:rgba(240, 240, 240, 100%);
}

QToolTip {
        border-radius: 4px;
        border: 1px solid rgba(155, 155, 155, 100%);
        background:rgba(240, 240, 240, 100%);
        padding: 2px;
        margin: 0px;
        color: #313639;
}

 

Where we are defining the values of property like background for LXQTPanel1 which is the primary panel but if we want to theme the second panel differently then we need to use LXQTPanel2 and if we require same theme then no need to add the property for second panel it will take the property of panel 1 only,

We can define appearance of things like tooltip using QToolTip , LXQtPanelPlugin for plugins area and property of widgets like calendar using QCalendarWidget like

QCalendarWidget #qt_calendar_navigationbar,
QCalendarWidget #qt_calendar_navigationbar * {
    background:rgba(240, 240, 240, 100%);
    color: rgba(54, 54, 54, 100%);
}
QCalendarWidget QToolButton {
    margin: 3px;
    border-radius: 4px;
    border: 1px solid transparent;
    padding: 4px;
} 

 

We can also add or change the icons like the default icon for LXQt main menu it the LXQT logo but we want it to be something else like the logo of Meilix. We can add the path of the icon in the QSS file like

#MainMenu {
        qproperty-icon: url(mainmenu.svg);
}

 

To check all the changes made in LXQT panel theme without restart or logout and login again we can use the following commands.

killall lxqt-panel
lxqt-panel

Resources

Deleting Meilix Github Releases

Meilix is the repository which uses build script to generate community version of lubuntu as LXQT Desktop. Meilix-Generator is the webapp which uses Meilix to generate ISO and deploy it on Meilix Github Release. Then the webapp mail the link of the ISO to the user.
Increasing number of ISO will increase the number of releases which results in dirty looking of Meilix repository. So we need to delete older releases after certain interval of time to make the repository release page looks good and decrease unwanted space.
This releases_maintainer.sh script will do this work for us.

#!/usr/bin/env bash
set -e
echo "This is a script to delete obsolete meilix iso builds by Abishek V Ashok"
echo "You have to add an authorization token to make it functional."

# jq is the JSON parser we will be using
sudo apt-get -y install jq

# Storing the response to a variable for future usage
response=`curl https://api.github.com/repos/fossasia/meilix/releases | jq '.[] | .id, .published_at'`

index=1  # when index is odd, $i contains id and when it is even $i contains published_date
delete=0 # Should we delete the release?
current_year=`date +%Y`  # Current year eg) 2001
current_month=`date +%m` # Current month eg) 2
current_day=`date +%d`   # Current date eg) 24

for i in $response; do
    if [ $((index % 2)) -eq 0 ]; then # We get the published_date of the release as $i's value here
        published_year=${i:1:4}
        published_month=${i:6:2}
        published_day=${i:9:2}

        if [ $published_year -lt $current_year ]; then
             let "delete=1"
        else
            if [ $published_month -lt $current_month ]; then
                let "delete=1"
            else
                if [ $((current_day-$published_day)) -gt 10 ]; then
                    let "delete=1"
                fi
            fi
        fi
    else # We get the id of the release as $i`s value here
        if [ $delete -eq 1 ]; then
            curl -X DELETE -H "Authorization: token $KEY" https://api.github.com/repos/fossasia/meilix/releases/$i
            let "delete=0"
        fi
    fi
    let "index+=1"
done

This code uses Github API to curl the Meilix releases. Github API is very useful in providing lots of information but here we are only concerned with the release date and time of the build.
Then we setup a condition if that satisfies then the release will automatically will get deleted.

For taking care of the authentication, a token has been uploaded to the Travis settings of Meilix of FOSSASIA.

The personal token has been generated by a user with write access to the repository with repo scope token.

This sort out the issue of having bulk of releases in the Meilix repository of FOSSASIA.

References:
Users Github API  by REST API v3
Repo Github API   by REST API v3

Wallpaper Strategy for Meilix Generator

We were first hosting the wallpaper uploaded by user on the Heroku server which was downloaded by the Travis CI which was previous solution for sending wallpaper to the Travis CI, but the problem was that wallpaper was downloaded only when build started building the ISO.

Downloading the hosted wallpaper was not a problem but the problem in that method was that the wallpaper hosted can be changed if another user also starts the build using Meilix Generator and uploads the wallpaper which will replace the previous wallpaper so simultaneous builds was not possible in previous method and resulted in conflicts.

So we thought of a sending the wallpaper to the Travis CI server for that we used base64 to and encoded it to a string using this.

with open(filename,'rb') as f:
    os.environ["Wallpaper"] = str(base64.b64encode(f.read()))[1:]

 

After uploading we send it as a variable to Travis CI and decode it. We will receive a binary file after decoding now we need to detect the mime type of the file and rename it accordingly before applying for that we use a script like this.

#renaming wallpaper according to extension png or jpg
for f in wallpaper; do
    type=$( file "$f" | grep -oP '\w+(?= image data)' )
    case $type in  
        PNG)  newext=png ;;
        JPEG) newext=jpg ;;
        *)    echo "??? what is this: $f"; continue ;;
    esac
    mv "$f" "${f%.*}.$newext"
done

 

After fixing the wallpaper extension we can apply it using the themes by replacing it with the theme wallpaper.

Resources

Base64 python documentation

How to Send a Script as Variable to the Meilix ISO with Travis and Meilix Generator

We wanted to add more features to Melix Generator web app to be able to customize Meilix ISO with more features so we thought of sending every customization we want to apply as a different variable and then use the scripts from Meilix Generator repo to generate ISO but that idea was bad as many variables are to be made and need to be maintained on both Heroku and Travis CI and keep growing with addition of features to web app.

So we thought of a better idea of creating a combined script with web app for each feature to be applied to ISO and send it as a variable to Travis CI.

Now another problem was how to send script as a variable after generating it as json do not support special characters inside the script. We tried escaping the special characters and the data was successfully sent to Travis CI and was shown in config but when setting that variable as an environment variable in Travis CI the whole value of variable was not taken as we had spaces in the script.

So to eliminate that problem we encoded the variable in the app as base64 and sent it to Travis CI and used it using following code.

To generate the variable from script.

with open('travis_script_1.sh','rb') as f:
    os.environ["TRAVIS_SCRIPT"] = str(base64.b64encode(f.read()))[1:]

 

For this we have to import base64 module and open the script generated in binary mode and using base64 we encode the script and using Travis CI API we send variable as script to the Travis CI to build the ISO with script in chroot we were also required to make changes in Meilix to be able to decode the script and then copy it into chroot during the ISO build.

sudo su <<EOF
echo "$TRAVIS_SCRIPT" > edit/meilix-generator.sh
mv browser.sh edit/browser.sh
EOF 

 

Using script inside chroot.

chmod +x meilix-generator.sh browser.sh
echo "$(<meilix-generator.sh)" #to test the file
./browser.sh
rm browser.sh

Resources

Base64 python documentation from docs.python.org

Base64 bash tutorial from scottlinux.com by Scott Miller

su in a script from unix.stackexchange.com answered by Ankit

How to customize LXQT for Meilix

We had a task of customizing the LXQT (LXQt is the Qt port and the upcoming version of LXDE, the Lightweight Desktop Environment) desktop environment in Meilix for events. For example, we can use a distro at events for presentations so during presentations things like system sounds, notifications and panel can be disturbing elements of presentations so we required LXQT to be pre configured for that so the time required in configuration is not wasted or none of the presentations are disturbed.

The default configuration of LXQT are present in ~/.config/lxqt. Configuration files are in This directory is initialized automatically. The default configuration for new users is found in /etc/xdg/lxqt but we are going to use skel for this so that if there are some changes in future in the LXQT code and they do not match default settings we can always be back to default settings by deleting user side changes. Similar to LXDE. LXQt provides a GUI applications to change its settings as well.

While Openbox is the default window manager we have used in Meilix with  LXQt, you can specify a different window manager to use with LXQt via by editing ~/.config/lxqt/session.conf. To a window manager of choice. Change the following line:

window_manager=openbox

 

We have used the Openbox for Meilix.

To create a configuration for panel we can go to ~/.config/lxqt and edit the panel.conf file if it is not present we can create a file named panel.conf and add/edit the code.

[panel1]
hidable=true 

 

GIF representing auto hide of panel.

In order to configure more things like notification we can edit the ~/.config/lxqt/notification.conf.

We can change things like the placement or the size of notification or the timeout of notification in this file.

For eg:

 [General]
__userfile__=true
placement=top-left
server_decides=1
spacing=6
width=295

 

We can now place all the configurations inside the skel folder so that every new user gets the same configurations we have made.

Resources

Updating of Linux Standard Base (lsb) and Shorten of log of Meilix Build

Updating the Linux Standard Base of the Meilix

Originally Meilix uses the Ubuntu mirror to fetch the Kernel source for the building of the Operating System. Therefore whenever a user type lsb_release -a for fetching the required information, they get the Ubuntu build info. The task is to change the config file to update the Linux Base Information from Ubuntu to Meilix.

lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 17.04
Release: 17.04
Codename: zesty

We need to patch a file in the location meilix-default-settings/etc/lsb-release which contains the information of the lsb release and this will overwrite the original configuration of Meilix.
This is how the lsb-release file looks like now:

DISTRIB_ID=Meilix
DISTRIB_RELEASE=17.04
DISTRIB_CODENAME=meilix
DISTRIB_DESCRIPTION="Meilix 17.04"

We made the required changes in the file as shown above and the output is as follows:

Shorten the log length of Meilix in Travis

We are facing issues while deployment of Meilix ISO in Travis and the error follows is ReadTimeout Error. Example log of one of fail build is:

This error gets solved automatically and the the ISO gets deployment after 1 or 2 restart of the build. But this time the error doesn’t get solved and after several attempts of restart of build, the ISO doesn’t get deployed.

Reason behind the error:

Travis is taking a lot of time to build the ISO. Travis logs are exceeding the time limit.

Proposed solution:

Reduce the time of build or shift to a new CI.
Reduce the log of the build so as to get the log within 9999 lines.

Solution Implemented

The best solution is to reduce the number of lines used in the log and this will also reduce the time of the build.
I tried concealing some command outputs by appending >/dev/null 2>&1 to some of the commands that has long outputs and adding -y to the commands like:

apt-get -qq -y --purge install file-roller unrar

References

Wiki of Linux Standard Base
Linux Foundation lsb
Ubuntu answer to reduce log