Author: xeon-zolt

This author has written 16 articles
Read more about the article Keep updating Build status in Meilix Generator

Keep updating Build status in Meilix Generator

One of the problems we faced while working Meilix Generator was to provide user with the status of the custom ISO build in the Meilix Generator web app so we came up with the idea of checking the status of the link generated by the web app. If the link is available the status code would be 200 otherwise it would be 404.

We have used python script for checking the status of URL. For generating URL, we use the tag name which will be used as a variable to generate the URL of the unique event user wants the ISO for and the date will help in generation of link rest of the link remains the same.

tag = os.environ["TRAVIS_TAG"]
date = datetime.datetime.now().strftime('%Y%m%d')
url=https://github.com/xeon-zolt/meilix/releases/download/"+tag+"/meilix-zesty-"+date+"-i386.iso"

 

Now we will use urllib for monitoring the status of link.

req = Request(url)
    try:
        response = urlopen(req)
    except HTTPError as e:
        return('Building Your Iso')
    except URLError as e:
        return('We failed to reach the server.')
    else:
        return('Build Sucessful : ' + url)

 

After monitoring the status the next step was to update the status dynamically on the status page.

So we’llll use a status function in the flask app which is used by JavaScript to get status of the link after intervals of time.

Flask :

@app.route('/now')
def status_url():
    return (status())

 

Javascript:

<script type ="text/javascript">
let url ="/now"
function getstatus(url)
{
    fetch(url).then(function(response){
        return response.text()
    }).then(function(text){
        console.log("status",text)
        document.querySelector("div#status")
        .innerHTML = text
    })
    }
window.onload = function(){
    fetch(url).then(function(response){
        return response.text()
    }).then(function(text){
        console.log("status",text)
        document.querySelector("div#status")
        .innerHTML = text
    })
    window.setInterval(getstatus.bind(null,url),30*1000)
}
/*setInterval(function,interval in millsecs)*/
</script>

 

This covers various steps to prompt user whether the build is ready or not.

Resource

Continue ReadingKeep updating Build status in Meilix Generator
Read more about the article How to make changes in Meilix Without rebuilding the ISO

How to make changes in Meilix Without rebuilding the ISO

We were building Meilix from build scripts from webapp which was taking 20 minutes approx. So to reduce that time we had an idea of using a pre built ISO as it requires fewer resources and less time as compared to the building the ISO from build script and makes modifications in it which would take less time after testing it took approx 8 minutes. The following steps were followed to edit Meilix ISO.

We require following packages for unpacking and repacking the ISO.

  • squashfs-tools
  • Genisoimage

Let’s start by unpacking the ISO. For that, we first mount the ISO.

sudo mount -o loop meilix-zesty-20170611-i386.iso mnt/

 

Now we extract the content of the ISO into a directory extract-cd and extract the squash file system and move it to edit folder to prepare chroot.

sudo rsync --exclude=/casper/filesystem.squashfs -a mnt/ extract-cd
sudo unsquashfs mnt/casper/filesystem.squashfs
sudo mv squashfs-root edit

 

Now we can chroot and do the editing we require to do in the ISO.

sudo mount -o bind /run/ edit/run
sudo cp /etc/hosts edit/etc/
sudo mount --bind /dev/ edit/dev
sudo chroot edit

 

After doing the changes in chroot. For doing changes we can make a separate script to be executed inside the chroot.

exit
EOF
sudo umount edit/dev

 

After completing all the changes we required in the ISO the important part comes that is repacking the ISO with the applied changes.

Regenerate the manifest.

sudo chmod +w extract-cd/casper/filesystem.manifest
sudo su <<HERE
chroot edit dpkg-query -W --showformat='${Package} ${Version}\n' > extract-cd/casper/filesystem.manifest <<EOF
exit
EOF
HERE
sudo cp extract-cd/casper/filesystem.manifest extract-cd/casper/filesystem.manifest-desktop
sudo sed -i '/ubiquity/d' extract-cd/casper/filesystem.manifest-desktop
sudo sed -i '/casper/d' extract-cd/casper/filesystem.manifest-desktop

 

Now we compress the file system we have just edited.
For higher compression we can increase the block size or use xz but that will increase the cost of compression time so we didn’t choose it for Meilix as we required a faster method.

sudo mksquashfs edit extract-cd/casper/filesystem.squashfs -noappend

 

Now we are going to calculate the MD5 sums again for the changes and replace them with the older MD5 sums.

cd extract-cd/ && find . -type f -not -name md5sum.txt -not -path '*/isolinux/*' -print0 | xargs -0 -- md5sum > md5sum.txt

 

Last step is to go in the edit directory and generate the ISO.

mkisofs \
    -V "Custom Meilix" \
    -r -cache-inodes -J -l \
    -b isolinux/isolinux.bin \
    -c isolinux/boot.cat \
    -no-emul-boot -boot-load-size 4 -boot-info-table \
    -o ../meilix-i386-custom.iso .

 

This covers all the steps need to make changes in Meilix without rebuilding ISO.

Resources:

Continue ReadingHow to make changes in Meilix Without rebuilding the ISO
Read more about the article Flask App to Upload Wallpaper On the Server for Meilix Generator

Flask App to Upload Wallpaper On the Server for Meilix Generator

We had a problem of getting a wallpaper from the user using Meilix Generator and use the wallpaper with the Meilix build scripts to generate the ISO. So, we were required to host the wallpaper on the server and downloaded by Travis CI during the build to include it in the ISO.

A solution is to render HTML templates and access data sent by POST using the request object from the flask. Redirect and url_for will be used to redirect the user once the upload is done and send_from_directory will help us to host the file under the /uploads that the user just uploaded which will be downloaded by the Travis for building the ISO.

We start by creating the HTML form marked with enctype=multipart/form-data.

<form action="upload" method="post" enctype="multipart/form-data">
        <input type="file" name="file"><br /><br />
        <input type="submit" value="Upload">
 </form>

 

First, we need imports of modules required. Most important is werkzeug.secure_filename().

import os
from flask import Flask, render_template, request, redirect, url_for, send_from_directory
from werkzeug import secure_file

 

Now, we’ll define where to upload and the type of file allowed for uploading. The path to upload directory on the server is defined by the extensions in app.config which is uploads/ here.

app.config['UPLOAD_FOLDER'] = 'uploads/'
app.config['ALLOWED_EXTENSIONS'] = set(['png', 'jpg', 'jpeg'])

 

This functions will check for valid extension for the wallpaper which are png, jpg and jpeg in this case defined above in app.config.

def allowed_file(filename):
    return '.' in filename and \
           filename.rsplit('.', 1)[1] in app.config['ALLOWED_EXTENSIONS']

 

After, getting the name of uploaded file from the user then using above function check if there are allowed file type and store it in a variable filename after that it move the files to the upload folder to save it.

Upload function check if the file name is safe and remove unsupported characters (line 3) after that moves it from a temporal folder to the upload folder. After moving, it renames the file as wallpaper so that the download link is same always which we have used in Meilix build script to download from server.

def upload():
    file = request.files['file']
    if file and allowed_file(file.filename):
        filename = secure_filename(file.filename)
        file.save(os.path.join(app.config['UPLOAD_FOLDER'], filename))
         os.rename(UPLOAD_FOLDER + filename, UPLOAD_FOLDER+'wallpaper')
         filename = 'wallpaper'

 

At this point, we have only uploaded the wallpaper and renamed the uploaded file to ‘wallpaper’ only. We cannot access the file outside the server it will result in 403 error so to make it available, the uploaded file need to be registered and then hosted using below code snippet.

We can also register uploaded_file as build_only rule and use the SharedDataMiddleware.

@app.route('/uploads/<filename>')
def uploaded_file(filename):
    return send_from_directory(app.config['UPLOAD_FOLDER'],filename)

The hosted wallpaper is used by Meilix in Travis CI to generate ISO using the download link which remains same for the uploaded wallpaper.

Why should we use secure secure_filename() function?

just imagine someone sends the following information as the filename to your app.

filename = "../../../../home/username/.sh"

 

If the number of ../ is correct and you would join this with your UPLOAD_FOLDER the hacker might have the ability to modify a file on the server’s filesystem that he or she should not modify.

Now, let’s look how the function works.

secure_filename('../../../../home/username/.sh')
'home_username_.sh'

Improving the uploads

We can add validation to the size of the file to be uploaded so that in case a user tries to upload a file too much big that may increase load on the server.

from flask import Flask, Request
app = Flask(__name__)
app.config['MAX_CONTENT_LENGTH'] = 16 * 1024 * 1024

Resources

Continue ReadingFlask App to Upload Wallpaper On the Server for Meilix Generator
Read more about the article How Meilix Generator sends Email Notifications with SendGrid

How Meilix Generator sends Email Notifications with SendGrid

We wanted to notify the users once the build was ready for download. To solve this we attempted making an email server on Meilix Generator but that can send email when it starts but it would take around 20 minutes to get the build ready so we thought of checking the deploy link status and send email whenever the link status was available (200) but the problem with this method was that the link can be pre available if ISO is rebuilt for same event.

Then, we attempted sending mail by Travis CI but the problem in that was closed SMTP ports (they have a strict policy about that) then we thought that Travis CI can trigger the Sendgrid which can send email to the user with the help of API.

We will use this code so that once the deployment of ISO by Travis CI is done it can execute the email script which requests Sendgrid to send email to the user.

after_deploy:
  - ./mail.py

 

We can create code using code generation service of Sendgrid we are going to choose python as it is easier to manipulate strings in python and we are going to use email as an environment variable.

After generation of python 3 code from the sendgrid website we are going to edit the message and email and hide the API key as an environment variable and create an authorization string to be used there too.

The URL will be generated by the below script as the body of url remains same only two things will change the TRAVIS_TAG which is event name and date.

date = datetime.datetime.now().strftime('%Y%m%d')
url="https://github.com/xeon-zolt/meilix/releases/download/"+os.environ["TRAVIS_TAG"]+"/meilix-zesty-"+date+"-i386.iso"

 

We can use this to hide the api key and use it as an environment variable because if the api key is visible in logs anyone can use it to exploit it and use it for spamming purpose.

authorization = "Bearer " + os.environ["mail_api_key"]
headers = {
    'authorization': authorization,

 

The main thing left to edit in the script is the message which is in the payload and is a string type so we are going to use the email received by Meilix generator as an environment variable and concatenate it with the payload string the message sent is in the value which is in the HTML format and we add the generated URL in similar way we added email variable to string.

payload = "{\"personalizations\":[{\"to\":[{\"email\":\"" + os.environ["email"] + "\"}],\"subject\":\"Your ISO is Ready\"}],\"from\":{\"email\":\"xeon.harsh@gmail.com\",\"name\":\"Meilix Generator\"},\"reply_to\":{\"email\":\"xeon.harsh@gmail.com\",\"name\":\"Meilix Generator\"},\"subject\":\"Your ISO is ready\",\"content\":[{\"type\":\"text/html\",\"value\":\"<html><p>Hi,<br>Your ISO is ready<br>URL : "+url+"<br><br>Thank You,<br>Meilix Generator Team</p></html>\"}]}"

 

The sent email looks like this

References

Continue ReadingHow Meilix Generator sends Email Notifications with SendGrid
Read more about the article Package Manager Translation for Meilix

Package Manager Translation for Meilix

There are many Linux distros and all of them use variety of different package managers. So a particular user of that specific Linux distro is familiar with that distro package manager commands only. Due to which when that user is out at a event or someplace else and require to install or remove or update package using the commands he is familiar with, he may get errors in doing so if that distro doesn’t have a package manager that he is familiar with.

To overcome this problem we can have a solution of adding package manager command translating functionality to Meilix. To translate the commands of package manager like pacman, apt, yum, zypper we have build translation modules for each. To install these modules we first check the Linux distro and map it to the package manager it is using. For this we write the following script.

 declare -A osInfo;
  osInfo[/etc/redhat-release]=yum
  osInfo[/etc/arch-release]=pacman
  osInfo[/etc/gentoo-release]=emerge
  osInfo[/etc/SuSE-release]=zypp
  osInfo[/etc/debian_version]=apt-get

 

Then after checking the native package manager it copy the modules required for that packet manger to the bin and makes them executable.These modules can be called by the names of the packet manager not available on

These modules can be called by the names of the packet manager not available on system. The module reads the arguments and convert command according to it. Like for pacman to apt module, a simple pacman command to install a app is

Now, the pacman is a module called from bin using two arguments and these two arguments use a switch statement are converted.

Example of commands in ubuntu / debian based system using apt but the user was familiar with pacman

Installing package:

pacman -S PACKAGE

Gets translated to:

apt install PACKAGE

Remove package:

apt install PACKAGE

Gets translated to:

apt remove PACKAGE

Update software database :

pacman -Syy

Gets translated to:

apt update

Show updatable packages:

pacman -Qu

Gets translated to:

apt list --upgradable

Update all:

pacman -Syu

Gets translated to:

apt upgrade

 

Mew ensures the cross distro package manager command compatibility by providing translations which is a helpful tool especially at events where users may find it difficult to operate system if he cannot install or add the specific package he requires at that time. Mew helps in making the user experience better as the user don’t have to struggle with the package manager commands he is not familiar with.

contribute to the project by forking: https://github.com/fossasia/mew

Continue ReadingPackage Manager Translation for Meilix
Read more about the article Arch Linux for Travis CI

Arch Linux for Travis CI

Travis CI does not provide Arch Linux as a testing environment but if you need to test some script or some app specifically on Arch Linux, you would have to do that manually for every change made into code and that would waste time. I had a similar problem when I was trying to test the Event Linux build scripts based on Arch Linux which required the Arch Linux environment to be able to run build scripts which take around 25 to 30 minutes to be tested for every change

How to use Arch travis script for Travis CI?

To solve the problem we need to use Arch as chroot in Travis CI. To do so we are going to use this script (link) which will deploy Arch Linux chroot on travis for us to start testing
For using it we need to configure the travis.yml like this :-

sudo: required
arch:
 repos:
  - archlinuxfr=http://repo.archlinux.fr/$arch
 packages:
  # pacman packages
  - yaourt
  - archiso
 script:

  - ./build-repo
  - sudo ./build.sh -v

script:
 - "travis_wait 30 sleep 1800 &"
 - curl -s https://raw.githubusercontent.com/fossasia/arch-travis/master/arch-travis.sh | bash

arch.packages defines a list of packages (from official repository or AUR) to be installed before running the build.
For eg:

packages:
  # pacman packages
  - python
  - perl
  # aur packages
  - go-git
  # packages from custom repo
  - yaourt

Here in arch.repos we define custom repository required for the package which is required during testing.

For eg. in above code repo : http://repo.archlinux.fr/$arch is used to install yaourt we could have used AUR too but that will increase our build testing time as the AUR will first download the pkgbuild and then build the package which can result in increase of time

arch.script defines a list of scripts to run as part of the build. Like in above example we have used ./build-repo and sudo ./build -v

Anything defined in the arch.script list will run from the base of the repository as a normal user called travis CI . sudo is available as well as any packages installed in the setup. The path of the build dir (or repository base), is stored in the TRAVIS_BUILD_DIR environment variable inside the chroot.

At the end of example we have used
script:
Which defines the scripts to be run by travis, this is where arch-travis was initialized and then we pipe it

Usage of travis-wait function

script:

- travis_wait 30 sleep 1800 &
 - curl -s https://raw.githubusercontent.com/fossasia/arch-travis/master/arch-travis.sh | bash

We cannot use travis wait function inside the chroot neither we can export it so we apply the travis wait function at the point where we have initialised the Arch-travis

Example usage for pkgbuild scripts

sudo: required # this to get sudo permissions

arch: # anything to be tested under Arch is defined below this
 script: 
  - makepkg app.pkgbuild # this will test the pkgbuild script

script: #for setting up arch chroot using the arch travis script
 - curl -s https://raw.githubusercontent.com/fossasia/arch-travis/master/arch-travis.sh | bash

Limitations

  • Increases build time by about 1-3 min.
  • Does not work on travis container-based infrastructure because sudo is required.
  • Limited configuration.
  • Does not include base group packages.
Continue ReadingArch Linux for Travis CI