Building 64 bit lzma file

Meilix contains the kernel and casper file in image-i386.tar.lzma. This lzma is a compression method for files. It provides a high compression ratio and fast decompression.

Lzma files contains the Meilix file system, we get the same content inside it which we get when when we mount an ISO file. Lzma file is basically an image of the Meilix Operating System.

Solution:

We need to build a lzma file of 64 bit for Meilix. If we extract the lzma file we will find a README file which contains all the information of the ISO used to build this lzma file.

When we search through the files we find a file named README.diskdefines which contains the information of the file system.

#define DISKNAME  Lubuntu 12.10 "Quantal Quetzal" - Release i386
#define TYPE  binary
#define TYPEbinary  1
#define ARCH  i386
#define ARCHi386  1
#define DISKNUM  1
#define DISKNUM1  1
#define TOTALNUM  0
#define TOTALNUM0  1

The first line depicts that name and version of the ISO with the architecture from where this file system has been taken.

So we thought to build the same system with release amd64.

We download the ISO and tried to compress it into lzma.tar.

Decompression of lzma.tar file is easy but to compress a file into lzma.tar is done as follows:

When using tar, the first word after -f is the output filename

tar -c --lzma -f foo.tar.lzma sourcefile(s)

We mounted the lubuntu 12.10 64 bit ISO and copied it into another folder. Then tried to compress it with lzma algorithm. It resulted in a big size file which can’t be pushed to Github.

So we find out that a file name filesystem.casper is largest in size and can also be removed. Therefore we removed that file and build the ISO again and pushed.
We tested the ISO and file that the build is working file.

Changes made:

  • One we need to push the 64 bit lzma
  • We need to edit this line in build.sh of Meilix

# Arch to build ISO for, i386 or amd64
arch=${1:-i386}

Finally we got the file as image-amd64.tar.lzma

So, if we look into the Readme file we can found the version which it used for building the zip.

#define DISKNAME  Ubuntu 12.10 "Quantal Quetzal" - Release amd64
#define TYPE  binary
#define TYPEbinary  1
#define ARCH  amd64
#define ARCHamd64  1
#define DISKNUM  1
#define DISKNUM1  1
#define TOTALNUM  0
#define TOTALNUM0  1

We can see that it is based on the same version but the architecture is different.

Resources

  1. GNU lzma tar: https://www.gnu.org/software/tar/manual/html_node/gzip.html
  2. Python lzma compression: https://docs.python.org/3/library/lzma.html

 

 

Continue ReadingBuilding 64 bit lzma file

Meilix Blank Screen

Meilix has recently faced a blank screen issue where while booting the system, it leads to a screen as follows:

We thought it to be virtual machine issue, and build the Meilix ISO in local system from scratch. But it still coming up with the same error.

Then we started to search for the commit which seems to be breaking the ISO. There is no such commit which actually breaks the ISO. So we can’t remove/revert to any particular commit.

Then we started searching in the history to search for the latest working ISO and found that this commit actually gives a working ISO 6f287ea and below this commit, none of the others are giving an ISO which works.

So we started to look into the particular commit to find the code which is breaking the system, but actually none of them actually seems to do so. It raises more unobvious issue.

If I searched for the error, it doesn’t seems to be error from the ISO side.Even I tried building the ISO locally and test it, it is giving the same result.

It looks like it’s more than a hardware error than the error on the iso side. I tried different qemu, virtualbox, VMware on different machines.

Another Idea tried for working of the ISO

  1. initramfs is a small linux shell. please use ls and cd to find the file containing the file system filesystem.squashfs. in this black shell (initramfs)
  2. try to mount the ISO file manually

I local mount using:

sudo mount -o loop /home/meilix.iso /home/user
sudo umount /home/user

 

I mount the March 18 (good) and the closest April (bad)l ISO image and run a diff between both directories. the result is possibly large.

Finally tabesin made a commit dce3b08 which resolves the issue. It just the removal of language pack.

We can find the ISO link: https://github.com/fossasia/meilix/releases/tag/untagged-e337428be18400673be6

References:

USB Boot Issue

Initramfs error

Continue ReadingMeilix Blank Screen

Using file.io for Meilix Deployment

Meilix is developed in FOSSASIA and is deployed as a release on Github, but the download speed on GitHub for large files is slow. Alternatively deployment can be done on 3rd party servers. transfer.sh offers a good service for a start, but they only have a reduced storage for heavy usage such as what is required for Meilix. file.io is a good alternative. file.io has API features which can be used in .travis.yml to deploy meilix. For the time being, we are deploying a generator branch to file.io

Changes made in the .travis.yml

We need to edit the .travis.yml to deploy the artifact on file.io

deploy:
provider: script
script: curl -F "file=@/home/travis/$(image_name)" https://file.io
on:
branch: generator

We need to edit the deploy attribute in the travis to get the deployment done in file.io. Query contains the address of the file which needs to be uploaded on file.io. Then the branch name is provided on which deployment needs to be done.

Output:

Travis executes the command to deploy the application.

{"success":true,"key":"5QBEry","link":"https://file.io/5QBEry","expiry":"14 days"}

success: true  the artifact is successfully deployed.

key and link: gives the link from where the ISO can be downloaded.

expiry: tell the number of days after which the ISO will be deleted and by default it is set to 14 days.

We can manually input expiry parameter to declare the expiry time.

script: curl -F "file=@/home/travis/$(image_name)" https://file.io/?expires=1w

This will set the expiry time to seven days. If you set it with w, it will be number of weeks, m will be for number of months, y will be number of years.

file.io solved the most important issue of meilix deployment and this approach can be use several different project of FOSSASIA for the deployment purpose.

References:

Continue ReadingUsing file.io for Meilix Deployment

Debuilding the meilix-default-settings Metapackage

In the Meilix code repository you find a metapackage named meilix-default-settings which contains custom settings in directories as debian, etc, and user. In these directories one can make changes to make them be included in the build ISO. As Meilix runs on Debian we package our custom user settings in a Debian package to be installed along all the other software packages. The process and utility to make a Debian package is called debuild.

Directories in the meilix-default-settings:

What is debuilding?

It’s Debian slang for “making a deb package” and that stirred quite some confusion in our communications. Debuild is actually a rebuilding of the metapackage. But as to rebuild the Debian package you usually type debuild -uc -us therefore I stick to the language

Suppose someone has edited a configuration file in the metapackage according to its desires to achieve a specific result in the ISO it won’t get in unless he rebuilds the metapackage.He has not only to edit the metapackage but also to rebuild it to get the desired output in the ISO. To make the process automated, we have made a tiny script which will debuild the metapackages during each and every build, we only need to modify the metapackage.

Actually the first meilix-default-settings folder is the only metapackage and inside of it is the sub-metapackage which is responsible to get the changes applied in the ISO. To see a change in the ISO, we only need to edit the meilix-default-settings usr or etc folder in the first layer. Then, we need to debuild the metapackages.

Code-Base:

This file is present here

1. #!/bin/bash
2. rm meilix-default-settings_*                                    
3. cd meilix-default-settings                                      
4. debuild -uc -us

Let’s go through the whole code base line by line:
Line 2 deletes the previous meilix-default-settings binary packages.
Line 3 in this we changed our directory to the metapackage folder that is of our concern.
Line 4 is the most important line, it builds the whole metapackage and brings back all the binary packages and metapackages after making the desired changes.

Follow the example below to know that actually how it works:

This pull request is responsible to turn off system sounds by default in the generated ISO. Pull Requests files in which I only edited the this file and rest of the files get changes in the process of debuilding the metapackage (ignore .travis.yml file).

References:
Required files under debian directory
Debian directory guideline

Continue ReadingDebuilding the meilix-default-settings Metapackage

Working of Meilix Build Script

Meilix build script is divided into several parts the initial part consist of getting the build environment ready for build which includes installation of packages to build the ISO.

Iike debootstrap genisoimage p7zip-full squashfs-tools ubuntu-dev-tools.

The we have a debuild script which is ran by the build script in initially to rebuild the metapackages so that if any change is created it can be included in the build.

To make the change in metapackage like meilix-default-settings.deb we include the changes in the meilix-default-settings/ to test the meta package locally we can use the following script to build it for testing.

#!/bin/bash
rm meilix-default-settings_*                                    #removes the older meilix-default-settings packages
cd meilix-default-settings                                      #cd into the metapackage directory
debuild -uc -us                                                 #debuild the meilix-default-settings metapackage

 

After creation of packages we prepare chroot with the help of debbootstrap using ubuntu mirror for application installation after that we copy the files we created using debuild script and the files we require in the chroot.
We have a different file for working in the chroot which executes once the chroot is ready.

Now we copy the kernel from chroot to the build server /image for creation of live cd

sudo cp chroot/boot/vmlinuz-**-generic image/casper/vmlinuz sudo cp chroot/boot/initrd.img-**-generic image/casper/initrd.lz

Next step is to extract initrd for updating the uuid information for the above changes.

7z e image/casper/initrd.lz && \
  mkdir initrd_FILES/ && \
  mv initrd initrd_FILES/ && \
  cd initrd_FILES/ && \
  cpio -id < initrd && \
  cd .. && \
  cp initrd_FILES/conf/uuid.conf image/.disk/casper-uuid-generic && \

Now we pack the ISO using mkisofs but the problem here is we don’t have a boot.cat file and without that we will have errors in packing the ISO so to handle that we first create a temporary ISO to be able to extract boot.cat from it.

IMAGE_NAME=${IMAGE_NAME:-"Meilix ${release} $(date -u +%Y%m%d) - ${arch}"}
ISOFILE=meilix-${release}-$(date -u +%Y%m%d)-${arch}.iso
sudo mkisofs -r -V "$IMAGE_NAME" -cache-inodes -J -l \
  -b isolinux/isolinux.bin -c isolinux/boot.cat \
  -no-emul-boot -boot-load-size 4 -boot-info-table \
  --publisher "Meilix Packaging Team" \
  --volset "Ubuntu Linux http://www.ubuntu.com" \
  -p "${DEBFULLNAME:-$USER} <${DEBEMAIL:-on host $(hostname --fqdn)}>" \
  -A "$IMAGE_NAME" \
  -m filesystem.squashfs \
  -o ../$ISOFILE.tmp .

After getting a updated boot.cat file by mounting the temporary ISO and extracting boot.cat from it, we repeat the similar steps to create the ISO.

Chroot Script

Inside the chroot script we are going to install all the packages we require in the ISO and the metapackages we created for ISO. we can also add the custom changes we require like turning off screen dimming or setting plymouth after installing the package.

echo -ne "\033[9;0]" >> /etc/issue
setterm -blank 0 >> /etc/issue

At the end of chroot script we do the cleanup.

perl -i -nle 'print unless /^Package: language-(pack|support)/ .. /^$/;' /var/lib/apt/extended_states 
apt-get -qq clean
 rm -rf /tmp/* #rm /etc/resolv.conf 
rm meilix-default-settings_1.0_all.deb 
rm meilix-metapackage_1.0-1_all.deb 
rm systemlock_0.1-1_all.deb plymouth-meilix-logo_1.0-1_all.deb plymouth-meilix-text_1.0-1_all.deb skype-ubuntu_4.1.0.20-1_i386.deb rm meilix-imclient_*_all.deb
rm /sbin/initctl dpkg-divert --rename --remove /sbin/initctl

Resources

Continue ReadingWorking of Meilix Build Script

Adding Manual ISO Controls in Phimpme Android

The Phimpme Android application comes with a well-featured camera to take high resolution photographs. It features an auto mode in the camera as well as a manual mode for users who likes to customise the camera experience according to their own liking. It provides the users to select from the range of ISO values supported by their devices with a manual mode to enhance the images in case the auto mode fails on certain circumstances such as low lighting conditions.

In this tutorial, I will be discussing how we achieved this in Phimpme Android with some code snippets and screenshots.

To provide the users with an option to select from the range of ISO values, the first thing we need to do is scan the phone for all the supported values of ISO and store it in an arraylist to be used to display later on. This can be done by the snippet provided below:

String iso_values = parameters.get("iso-values");
if( iso_values == null ) {
 iso_values = parameters.get("iso-mode-values"); // Galaxy Nexus
 if( iso_values == null ) {
    iso_values = parameters.get("iso-speed-values"); // Micromax A101
    if( iso_values == null )
       iso_values = parameters.get("nv-picture-iso-values"); // LG dual P990

Every device supports a different set of keyword to provide the list of ISO values. Hence, we have tried to add every possible keywords to extract the values. Some of the keywords used above covers almost 90% of the android devices and gets the set of ISO values successfully.

For the devices which supports the ISO values but doesn’t provide the keyword to extract the ISO values, we can provide the standard list of ISO values manually using the code snippet provided below:

values.add("200");
values.add("400");
values.add("800");
values.add("1600");

After extracting the set of ISO values, we need to create a list to display to the user and upon selection of the particular ISO value as depicted in the Phimpme camera screenshot below

Now to set the selected ISO value, we first need to get the ISO key to set the ISO values as depicted in the code snippet provided below:

if( parameters.get(iso_key) == null ) {
 iso_key = "iso-speed"; // Micromax A101
 if( parameters.get(iso_key) == null ) {
    iso_key = "nv-picture-iso"; // LG dual P990
    if( parameters.get(iso_key) == null ) {
       if ( Build.MODEL.contains("Z00") )
          iso_key = "iso"; // Asus Zenfone 2 Z00A and Z008

Getting the key to set the ISO values is similar to getting the key to extract the ISO values from the device. The above listed ISO keys to set the values covers most of the devices.

Now after we have got the ISO key, we need to change the camera parameter to reflect the selected change.

parameters.set(iso_key, supported_values.selected_value);
setCameraParameters(parameters);

To get the full source code on how to set the ISO values manually, please refer to the Phimpme Android repository.

Resources

  1. Stackoverflow – Keywords to extract ISO values from the device: http://stackoverflow.com/questions/2978095/android-camera-api-iso-setting
  2. Open camera Android source code: https://sourceforge.net/p/opencamera/code/ci/master/tree/
  3. Blog – Learn more about ISO values in photography: https://photographylife.com/what-is-iso-in-photography
Continue ReadingAdding Manual ISO Controls in Phimpme Android

How to make changes in Meilix Without rebuilding the ISO

We were building Meilix from build scripts from webapp which was taking 20 minutes approx. So to reduce that time we had an idea of using a pre built ISO as it requires fewer resources and less time as compared to the building the ISO from build script and makes modifications in it which would take less time after testing it took approx 8 minutes. The following steps were followed to edit Meilix ISO.

We require following packages for unpacking and repacking the ISO.

  • squashfs-tools
  • Genisoimage

Let’s start by unpacking the ISO. For that, we first mount the ISO.

sudo mount -o loop meilix-zesty-20170611-i386.iso mnt/

 

Now we extract the content of the ISO into a directory extract-cd and extract the squash file system and move it to edit folder to prepare chroot.

sudo rsync --exclude=/casper/filesystem.squashfs -a mnt/ extract-cd
sudo unsquashfs mnt/casper/filesystem.squashfs
sudo mv squashfs-root edit

 

Now we can chroot and do the editing we require to do in the ISO.

sudo mount -o bind /run/ edit/run
sudo cp /etc/hosts edit/etc/
sudo mount --bind /dev/ edit/dev
sudo chroot edit

 

After doing the changes in chroot. For doing changes we can make a separate script to be executed inside the chroot.

exit
EOF
sudo umount edit/dev

 

After completing all the changes we required in the ISO the important part comes that is repacking the ISO with the applied changes.

Regenerate the manifest.

sudo chmod +w extract-cd/casper/filesystem.manifest
sudo su <<HERE
chroot edit dpkg-query -W --showformat='${Package} ${Version}\n' > extract-cd/casper/filesystem.manifest <<EOF
exit
EOF
HERE
sudo cp extract-cd/casper/filesystem.manifest extract-cd/casper/filesystem.manifest-desktop
sudo sed -i '/ubiquity/d' extract-cd/casper/filesystem.manifest-desktop
sudo sed -i '/casper/d' extract-cd/casper/filesystem.manifest-desktop

 

Now we compress the file system we have just edited.
For higher compression we can increase the block size or use xz but that will increase the cost of compression time so we didn’t choose it for Meilix as we required a faster method.

sudo mksquashfs edit extract-cd/casper/filesystem.squashfs -noappend

 

Now we are going to calculate the MD5 sums again for the changes and replace them with the older MD5 sums.

cd extract-cd/ && find . -type f -not -name md5sum.txt -not -path '*/isolinux/*' -print0 | xargs -0 -- md5sum > md5sum.txt

 

Last step is to go in the edit directory and generate the ISO.

mkisofs \
    -V "Custom Meilix" \
    -r -cache-inodes -J -l \
    -b isolinux/isolinux.bin \
    -c isolinux/boot.cat \
    -no-emul-boot -boot-load-size 4 -boot-info-table \
    -o ../meilix-i386-custom.iso .

 

This covers all the steps need to make changes in Meilix without rebuilding ISO.

Resources:

Continue ReadingHow to make changes in Meilix Without rebuilding the ISO