Linux

Read more about the article Reducing the YouTube response time by 90%

Reducing the YouTube response time by 90%

In this blog post, we are going to cover how the audio from Youtube is being used in SUSI Smart Speaker and how we reduced the response time from ~40 seconds to ~4 seconds for an average music video length. First Approach Earlier, we were using MPV player’s inbuilt feature to fetch the YouTube music. However, MPV player was a bulky option and the music server had to be started every time before initiating a music video. video_process = subprocess.Popen(['mpv', '--no-video', 'https://www.youtube.com/watch?v=' + video_url[4:], '--really-quiet']) # nosec #pylint-disable type: ignore requests.get('http://localhost:7070/song/' + video_url) self.video_process = video_process stopAction.run() stopAction.detector.terminate() Making it Efficient To reduce the response time, we created a custom Music Server based on Flask,python-vlc and python-pafy which accepts requests from the main client and instructs the System to play the music with just 90% more efficiency. app = Flask(__name__) Instance = vlc.Instance('--no-video') player = Instance.media_player_new() url = '' @app.route('/song', methods=['GET']) def youtube():     vid = request.args.get('vid')     url = 'https://www.youtube.com/watch?v=' + vid     video = pafy.new(url)     streams = video.audiostreams      best = streams[3]     playurl = best.url     Media = Instance.media_new(playurl)     Media.get_mrl()     player.set_media(Media)     player.play()     display_message = {"song":"started"}     resp = jsonify(display_message)     resp.status_code = 200     return resp However, shifting to this Server removed the ability to process multiple queries and hence we were unable to pause/play/stop the music until it completed the time duration. We wanted to retain the ability to have ‘play/pause/stop’ actions without implementing multiprocessing or multithreading as it would’ve required extensive testing to successfully implement them without creating deadlocks and would’ve been overkill for a simple feature. Bringing Back the Lost Functionalities The first Step we took was to remove the vlc-python module and implement a way to obtain an URL that we use in another asynchronous music player. @app.route('/song', methods=['GET']) def youtube():     vid = request.args.get('vid')     streams = video.audiostreams     best = streams[3]     playurl = best.url      display_message = {"song": "started", "url": playurl}     resp = jsonify(display_message)     resp.status_code = 200     return resp The next issue was to actually find a way to run the Music Player asynchronously. We used the `subprocess. Popen` method and cvlc to play the songs asynchronously. try:     x = requests.get('http://localhost:7070/song?vid=' + video_url[4:])     data = x.json()     url = data['url']     video_process = subprocess.Popen(['cvlc', 'http' + url[5:], '--no-video'])     self.video_process = video_process except Exception as e:     logger.error(e); And this is how we were able to increase the efficiency of the music player while maintaining the functionalities. References https://helpmanual.io/help/cvlc/https://docs.python.org/3/library/subprocess.htmlhttps://pypi.org/project/pafy/

Continue ReadingReducing the YouTube response time by 90%
Read more about the article Speeding up the Travis Build to Decrease the Building Time

Speeding up the Travis Build to Decrease the Building Time

Meilix is the repository which uses build script to generate community version of lubuntu as LXQT Desktop. It usually takes around 25-26 to build and deploy the ISO as a Github Release on master branch. Observing the build log we can see that there are most of the packages like debootstrap, squashfs-tool, etc which are being fetch and setup at the time of building which results in increasing build time. The issue is to decrease the build time supplying the packages from Travis so that when we run build.sh we won't be required to download them again and thus few minutes will get reduced. We included list of packages to be pre-downloaded in .travis.yml include: - os: linux addons: apt: sources: - ubuntu-toolchain-r-test packages: - debootstrap - genisoimage - p7zip-full - squashfs-tools - ubuntu-dev-tools - dpkg-dev - debhelper - fakeroot - devscripts These are some important packages included in the build.sh  as devtools="debootstrap genisoimage p7zip-full squashfs-tools ubuntu-dev-tools" which are always fetched, so we included it into .travis.yml and downloaded it before entering into the chroot environment. By specifying those packages in the .travis.yml, Travis provides those packages beforehand into the docker container so it will run our script. Since the scripts also include package then when the script runs apt-get it won't download those packages again. They are specified outside the chroot environment because they are expected to be at the system the build.sh script is run to get the iso. By this way, we get a sharp decrease in build time as the internet in the Travis CI container is not so fast so the package download time can be avoided. Now the build is taking around 15-16 minutes to build and deploy. One thing to note that we didn’t remove those packages off build.sh so that build.sh works outside Travis CI as well. References: Pull #176 by @abishekvashok Speeding up Travis Build by Travis CI Faster Build by atchai.com

Continue ReadingSpeeding up the Travis Build to Decrease the Building Time
Read more about the article Setting up SUSI Desktop Locally for Development and Using Webview Tag and Adding Event Listeners
SUSI Desktop

Setting up SUSI Desktop Locally for Development and Using Webview Tag and Adding Event Listeners

SUSI Desktop is a cross platform desktop application based on electron which presently uses chat.susi.ai as a submodule and allows the users to interact with susi right from their desktop. Any electron app essentially comprises of the following components Main Process (Managing windows and other interactions with the operating system) Renderer Process (Manage the view inside the BrowserWindow) Steps to setup development environment Clone the repo locally. $ git clone https://github.com/fossasia/susi_desktop.git $ cd susi_desktop Install the dependencies listed in package.json file. $ npm install Start the app using the start script. $ npm start Structure of the project The project was restructured to ensure that the working environment of the Main and Renderer processes are separate which makes the codebase easier to read and debug, this is how the current project is structured. The root directory of the project contains another directory ‘app’ which contains our electron application. Then we have a package.json which contains the information about the project and the modules required for building the project and then there are other github helper files. Inside the app directory- Main - Files for managing the main process of the app Renderer - Files for managing the renderer process of the app Resources - Icons for the app and the tray/media files Webview Tag Display external web content in an isolated frame and process, this is used to load chat.susi.ai in a BrowserWindow as <webview src="https://chat.susi.ai/"></webview> Adding event listeners to the app Various electron APIs were used to give a native feel to the application. Send focus to the window WebContents on focussing the app window. win.on('focus', () => { win.webContents.send('focus'); }); Display the window only once the DOM has completely loaded. const page = mainWindow.webContents; ... page.on('dom-ready', () => { mainWindow.show(); }); Display the window on ‘ready-to-show’ event win.once('ready-to-show', () => { win.show(); }); Resources 1. A quick article to understand electron’s main and renderer process by Cameron Nokes at Medium link 2. Official documentation about the webview tag at https://electron.atom.io/docs/api/webview-tag/ 3. Read more about electron processes at https://electronjs.org/docs/glossary#process 4. SUSI Desktop repository at https://github.com/fossasia/susi_desktop.

Continue ReadingSetting up SUSI Desktop Locally for Development and Using Webview Tag and Adding Event Listeners
Read more about the article Enhancing SUSI Desktop to Display a Loading Animation and Auto-Hide Menu Bar by Default

Enhancing SUSI Desktop to Display a Loading Animation and Auto-Hide Menu Bar by Default

SUSI Desktop is a cross platform desktop application based on electron which presently uses chat.susi.ai as a submodule and allows the users to interact with susi right from their desktop. The benefits of using chat.susi.ai as a submodule is that it inherits all the features that the webapp offers and thus serves them in a nicely build native application. Display a loading animation during DOM load. Electron apps should give a native feel, rather than feeling like they are just rendering some DOM, it would be great if we display a loading animation while the web content is actually loading, as depicted in the gif below is how I implemented that. Electron provides a nice, easy to use API for handling BrowserWindow, WebContent events. I read through the official docs and came up with a simple solution for this, as depicted in the below snippet. onload = function () { const webview = document.querySelector('webview'); const loading = document.querySelector('#loading'); function onStopLoad() { loading.classList.add('hide'); } function onStartLoad() { loading.classList.remove('hide'); } webview.addEventListener('did-stop-loading', onStopLoad); webview.addEventListener('did-start-loading', onStartLoad); }; Hiding menu bar as default Menu bars are useful, but are annoying since they take up space in main window, so I hid them by default and users can toggle their display on pressing the Alt key at any point of time, I used the autoHideMenuBar property of BrowserWindow class while creating an object to achieve this. const win = new BrowserWindow({ … show: false, autoHideMenuBar: true }); Resources 1. More information about BrowserWindow class in the official documentation at electron.atom.io. 2. Follow a quick tutorial to kickstart creating apps with electron at https://www.youtube.com/watch?v=jKzBJAowmGg. 3. SUSI Desktop repository at https://github.com/fossasia/susi_desktop.

Continue ReadingEnhancing SUSI Desktop to Display a Loading Animation and Auto-Hide Menu Bar by Default
Read more about the article Functionality and Customization of the Meilix Metapackage meilix-default-settings

Functionality and Customization of the Meilix Metapackage meilix-default-settings

Meilix has is made of build file and metapackages. Build file is responsible for executing commands and successfully implementing the work of metapackages. Metapackages in Meilix Name of metapackages used in Meilix are: meilix-artwork, meilix-default-settings. meilix-default-settings meilix-default-settings have 3 major folders debian, etc and usr and a Makefile. We are only concerned with etc and usr folder here. etc and usr folders are folders in which if changes are made that can be seen the ISO. One can assume this as two folders present in the root folder of a Linux Distro. Its directory is like this: meilix-artwork meilix-artwork has 1 main folder named as usr which contain share folder in which plymouth configuration is made. One can make changes here and it will directly seen in the Linux Distro. Its directory looks like this: How these meta packages actually work? To get the answer one has to jump into the debian folder of any of the metapackage. It contains a control file. This contains information of the metapackages. Source: meilix-default-settings Section: x11 Priority: extra Maintainer: meilix <vanhonit@gmail.com> Build-Depends: debhelper (>= 8.0.0) Standards-Version: 3.9.2 Homepage: http://mbm.vn Package: meilix-default-settings Architecture: all Depends: ${shlibs:Depends}, ${misc:Depends} Description: default settings for meilix Various system settings tailored for meilix. One can update the metapackage from here and tweak with its depends. One come to know about the maintainer of the metapackage which can contacted in case of any issue. We can also know for which architecture this metapackage is made and about its description. The whole debian does the work but after making any changes in the metapackage, it needs to be rebuild which is performed by debuild.sh. This is how a metapackages in Meilix works. References: Linux Metapackages - Matthartley from linux.com Creating a Metapackage - Ajmitch from askubuntu.com

Continue ReadingFunctionality and Customization of the Meilix Metapackage meilix-default-settings
Read more about the article Updating of Linux Standard Base (lsb) and Shorten of log of Meilix Build

Updating of Linux Standard Base (lsb) and Shorten of log of Meilix Build

Updating the Linux Standard Base of the Meilix Originally Meilix uses the Ubuntu mirror to fetch the Kernel source for the building of the Operating System. Therefore whenever a user type lsb_release -a for fetching the required information, they get the Ubuntu build info. The task is to change the config file to update the Linux Base Information from Ubuntu to Meilix. lsb_release -a No LSB modules are available. Distributor ID: Ubuntu Description: Ubuntu 17.04 Release: 17.04 Codename: zesty We need to patch a file in the location meilix-default-settings/etc/lsb-release which contains the information of the lsb release and this will overwrite the original configuration of Meilix. This is how the lsb-release file looks like now: DISTRIB_ID=Meilix DISTRIB_RELEASE=17.04 DISTRIB_CODENAME=meilix DISTRIB_DESCRIPTION="Meilix 17.04" We made the required changes in the file as shown above and the output is as follows: Shorten the log length of Meilix in Travis We are facing issues while deployment of Meilix ISO in Travis and the error follows is ReadTimeout Error. Example log of one of fail build is: This error gets solved automatically and the the ISO gets deployment after 1 or 2 restart of the build. But this time the error doesn’t get solved and after several attempts of restart of build, the ISO doesn’t get deployed. Reason behind the error: Travis is taking a lot of time to build the ISO. Travis logs are exceeding the time limit. Proposed solution: Reduce the time of build or shift to a new CI. Reduce the log of the build so as to get the log within 9999 lines. Solution Implemented The best solution is to reduce the number of lines used in the log and this will also reduce the time of the build. I tried concealing some command outputs by appending >/dev/null 2>&1 to some of the commands that has long outputs and adding -y to the commands like: apt-get -qq -y --purge install file-roller unrar References Wiki of Linux Standard Base Linux Foundation lsb Ubuntu answer to reduce log

Continue ReadingUpdating of Linux Standard Base (lsb) and Shorten of log of Meilix Build
Read more about the article Configuration for Auto-hiding Panel in Meilix with LXQt desktop

Configuration for Auto-hiding Panel in Meilix with LXQt desktop

In the new LXQt desktop, we can intelligently hide the panel. For that purpose, we’ll just need to patch a new file in a location under the meilix-default-settings metapackage. Originally the file lies in the lxqt folder of the .config of the OS with the name panel.conf like .config/lxqt/panel.conf but since we have to make changes in the metapackage, we need to patch it here meilix-default-settings/etc/skel/.config/lxqt/panel.conf. Files in etc/skel/ will be put in each users’ new home folder, a folder which does not exist yet when we build the ISO. panel.conf 1. [panel1] 2. alignment=-1 3. animation-duration=0 4. background-color=@Variant(\0\0\0\x43\0\xff\xff\0\0\0\0\0\0\0\0) 5. background-image= 6. desktop=0 7. font-color=@Variant(\0\0\0\x43\0\xff\xff\0\0\0\0\0\0\0\0) 8. hidable=true 9. iconSize=22 10. lineCount=1 11. lockPanel=false 12. opacity=100 13.panelSize=32 14. plugins=mainmenu, desktopswitch, quicklaunch, taskbar, tray, statusnotifier, mount, volume, clock, showdesktop 15. position=Bottom 16. show-delay=0 17. width=100 18. width-percent=true In the line number 8 , hidable=true is doing all the jobs. It is the only line which hides the panel by default. How we find this approach? Originally LXQt panel is not hidden, they are shown by default. I first try to locate panel.conf file which will carry out the configuration for the panel. I try to find the code responsible for hiding the panel, but I can’t find that. Then I copied the panel.conf in a file and then by GUI I hide the panel and reopen the config file. Then I compare the changes between this file and the old config.panel file in which I found that the new file has a new line hidable=true. We introduced the changes in this PR. How this approach actually work? We are using meilix-default-settings metapackage to make the things work. We made an .config file which contains the configuration file. And the .config file is present under skel folder which gets copied under the home folder of the user. Thus ultimately we get a configuration file which will overwrite the original one to get the desired changes. Other Uses of panel.conf The file panel.conf could be used to customize all related settings to the LXQT panel, like its alignment, volume bar, quick launch, show desktop, etc. References: LXQt panel hiding Customize LXQt desktop

Continue ReadingConfiguration for Auto-hiding Panel in Meilix with LXQt desktop
Read more about the article Debuilding the meilix-default-settings Metapackage

Debuilding the meilix-default-settings Metapackage

In the Meilix code repository you find a metapackage named meilix-default-settings which contains custom settings in directories as debian, etc, and user. In these directories one can make changes to make them be included in the build ISO. As Meilix runs on Debian we package our custom user settings in a Debian package to be installed along all the other software packages. The process and utility to make a Debian package is called debuild. Directories in the meilix-default-settings: What is debuilding? It’s Debian slang for “making a deb package” and that stirred quite some confusion in our communications. Debuild is actually a rebuilding of the metapackage. But as to rebuild the Debian package you usually type debuild -uc -us therefore I stick to the language Suppose someone has edited a configuration file in the metapackage according to its desires to achieve a specific result in the ISO it won’t get in unless he rebuilds the metapackage.He has not only to edit the metapackage but also to rebuild it to get the desired output in the ISO. To make the process automated, we have made a tiny script which will debuild the metapackages during each and every build, we only need to modify the metapackage. Actually the first meilix-default-settings folder is the only metapackage and inside of it is the sub-metapackage which is responsible to get the changes applied in the ISO. To see a change in the ISO, we only need to edit the meilix-default-settings usr or etc folder in the first layer. Then, we need to debuild the metapackages. Code-Base: This file is present here 1. #!/bin/bash 2. rm meilix-default-settings_* 3. cd meilix-default-settings 4. debuild -uc -us Let’s go through the whole code base line by line: Line 2 deletes the previous meilix-default-settings binary packages. Line 3 in this we changed our directory to the metapackage folder that is of our concern. Line 4 is the most important line, it builds the whole metapackage and brings back all the binary packages and metapackages after making the desired changes. Follow the example below to know that actually how it works: This pull request is responsible to turn off system sounds by default in the generated ISO. Pull Requests files in which I only edited the this file and rest of the files get changes in the process of debuilding the metapackage (ignore .travis.yml file). References: Required files under debian directory Debian directory guideline

Continue ReadingDebuilding the meilix-default-settings Metapackage
Read more about the article How to make changes in Meilix Without rebuilding the ISO

How to make changes in Meilix Without rebuilding the ISO

We were building Meilix from build scripts from webapp which was taking 20 minutes approx. So to reduce that time we had an idea of using a pre built ISO as it requires fewer resources and less time as compared to the building the ISO from build script and makes modifications in it which would take less time after testing it took approx 8 minutes. The following steps were followed to edit Meilix ISO. We require following packages for unpacking and repacking the ISO. squashfs-tools Genisoimage Let's start by unpacking the ISO. For that, we first mount the ISO. sudo mount -o loop meilix-zesty-20170611-i386.iso mnt/   Now we extract the content of the ISO into a directory extract-cd and extract the squash file system and move it to edit folder to prepare chroot. sudo rsync --exclude=/casper/filesystem.squashfs -a mnt/ extract-cd sudo unsquashfs mnt/casper/filesystem.squashfs sudo mv squashfs-root edit   Now we can chroot and do the editing we require to do in the ISO. sudo mount -o bind /run/ edit/run sudo cp /etc/hosts edit/etc/ sudo mount --bind /dev/ edit/dev sudo chroot edit   After doing the changes in chroot. For doing changes we can make a separate script to be executed inside the chroot. exit EOF sudo umount edit/dev   After completing all the changes we required in the ISO the important part comes that is repacking the ISO with the applied changes. Regenerate the manifest. sudo chmod +w extract-cd/casper/filesystem.manifest sudo su <<HERE chroot edit dpkg-query -W --showformat='${Package} ${Version}\n' > extract-cd/casper/filesystem.manifest <<EOF exit EOF HERE sudo cp extract-cd/casper/filesystem.manifest extract-cd/casper/filesystem.manifest-desktop sudo sed -i '/ubiquity/d' extract-cd/casper/filesystem.manifest-desktop sudo sed -i '/casper/d' extract-cd/casper/filesystem.manifest-desktop   Now we compress the file system we have just edited. For higher compression we can increase the block size or use xz but that will increase the cost of compression time so we didn’t choose it for Meilix as we required a faster method. sudo mksquashfs edit extract-cd/casper/filesystem.squashfs -noappend   Now we are going to calculate the MD5 sums again for the changes and replace them with the older MD5 sums. cd extract-cd/ && find . -type f -not -name md5sum.txt -not -path '*/isolinux/*' -print0 | xargs -0 -- md5sum > md5sum.txt   Last step is to go in the edit directory and generate the ISO. mkisofs \ -V "Custom Meilix" \ -r -cache-inodes -J -l \ -b isolinux/isolinux.bin \ -c isolinux/boot.cat \ -no-emul-boot -boot-load-size 4 -boot-info-table \ -o ../meilix-i386-custom.iso .   This covers all the steps need to make changes in Meilix without rebuilding ISO. Resources: Ubuntu.com reference for Live CD Customization

Continue ReadingHow to make changes in Meilix Without rebuilding the ISO

Sending e-mail from linux terminal

So while finalizing the apk-generator for my GSoC project, I faced a roadblock in sending the generated App to the organizer. Normally the build takes around 10–12 minutes, so asking the user to wait for that long on the website and then providing him/her with a download link did not feel like a good option. (amirite?) So I and Manan Wason thought of a different approach to this problem, which was to email the generated app to the organzier. For doing this, we used 2 handy tools MSMTP and Mutt. We can use MSMTP to send email but unfortunately we cannot include attachments, so we used Mutt to help us send email with attachments from the command line. So hang tight and follow the rest of the guide to start sending emails from your terminal and get yourself some developer #swag Step 1 : Installation Use the following commands to install MSMTP and Mutt sudo apt-get -y install msmtp sudo apt-get -y install ca-certificates sudo apt-get -y install mutt We need to have a file that contains Certificate Authority (CA) certificates so that we can connect using SSL / TLS to the email server. Step 2 : Configuring MSMTP Now we’ll MSMTP configuration on /etc/msmtprc with the content below. NOTE : You will have to enter your username and password on this file so make sure to make this file private. Lets first open this file nano /etc/msmtprc Next, add following text to the file, account default tls on tls_starttls off tls_certcheck off auth on host smtp.mail.yahoo.com (change this to smtp.gmail.com for gmail) user username password password from username@yahoo.com logfile /var/log/msmtp.log NOTE : Refrain from using gmail as they might terminate your account for sending email via MSMTP. For configuring mutt, we’ll use a similar command to edit the file located at /root/.muttrc nano /root/.muttrc Add following text to it which specifies the MSMTP account to use for sending email set sendmail=”/usr/bin/msmtp” set use_from=yes set realname=”MY Real Name” set from=username@yahoo.com set envelope_from=yes That’s it! Now lets get ready for the fun part, SENDING THOSE EMAILS :D Step 3 : Sending Now, there are 2 cases that might arise while sending the email, 1 : Sending without attachment This is pretty straightforward and can be done with either MSMTP or Mutt. Using MSMTP printf “To: recipient@domain.comnFrom: username@domain.comnSubject: Testing MSMTPnnHello there. This is email test from MSMTP.” | msmtp recipient@domain.com Entering the following code will send the email to the recipient and also display the sent email in the terminal. Using Mutt mutt -s “Testing Mutt” — recipient@domain.com < /path/to/body.txt NOTE : ‘body.txt’ is the file whose contents will be used in the body of the email that will be sent to ‘recipient@domain.com’. 2 : Sending WITH an attachment Unlike the previous case, this can be done ONLY using Mutt and the code used is mutt -a /path/to/attachment.txt -s “Testing Mutt — recipient@domain.com < /path/to/body.txt The syntax is similar to the above case where we sent the email without attachments. So well, that it then! If you followed the instructions carefully, you will have a…

Continue ReadingSending e-mail from linux terminal