Adding swap space to your DigitalOcean droplet, if you run out of RAM

The Open Event Android App generator runs on a DigitalOcean. The deployment runs on a USD 10 box, that has 1 GB of RAM, but for testing I often use a USD 5 box, that has only 512mb of RAM.

When trying to build an android app using gradle and Java 8, there could be an issue where you run out of RAM (especially if it’s 512 only).

What we can do to remedy this problem is creating a swapfile. On an SSD based system, Swap spaces work almost as fast as RAM, because SSDs have very high R/W speeds.

Check hard disk space availability using

df -h

There should be an output like this

Filesystem      Size  Used Avail Use% Mounted on
udev            238M     0  238M   0% /dev
tmpfs            49M  624K   49M   2% /run
/dev/vda1        20G  1.1G   18G   6% /
tmpfs           245M     0  245M   0% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs           245M     0  245M   0% /sys/fs/cgroup
tmpfs            49M     0   49M   0% /run/user/1001

The steps to create a swap file and allocating it as swap are

sudo fallocate -l 1G /swapfile
sudo chmod 600 /swapfile
sudo mkswap /swapfile
sudo swapon /swapfile

We can verify using

sudo swapon --show
NAME      TYPE  SIZE USED PRIO
/swapfile file 1024M   0B   -1

And now if we see RAM usage using free -h , we’ll see

              total        used        free      shared  buff/cache   available
Mem:           488M         37M         96M        652K        354M        425M
Swap:          1.0G          0B        1.0G

Do not use this as a permanent measure for any SSD based filesystem. It can corrupt your SSD if used as swap for long. We use this only for short periods of time to help us build android apks on low ram systems.

Getting fired up with Firebase Database

As you might’ve noticed, in my Open Event Android Project, we are asking the user to enter his/her details and then using these details at the backend for generating the app according to his/her needs.

One thing to wonder is how did we transmit the details from webpage to the server.

Well, this is where Firebase comes to the rescue one more time!

If you’ve read my previous post on Firebase Storage, you might have started to appreciate what an awesometastic service Firebase is.

So without any further adieu, lets get started with this.

Step 1 :

Add your project to Firebase from the console.

 newProj
Click on the Blue button

Step 2 :

Add Firebase to your webapp

Open the project, you’ve just created and click on the bright red button that says, “ Add Firebase to your web app”

 addFirebase

Copy the contents from here and paste it after your HTML code.

Step 3 :

Next up, navigate to the Database section in your console and move to the Rules tab.

 screenshot-area-2016-07-18-204133.png

For now, let us edit the rules to allow anyone to read and write to the database.

 screenshot-area-2016-07-18-204437

Almost all set up now.

Step 4 :

Modify the HTML to allow entering data by the user

This looks something like this :

Now let us setup our javascript to extract this data and store this in Firebase Database.

We are almost finished with uploading the data to the database.

Enter data inside the fields and press submit.

If everything went well, you will be able to see the newly entered data inside your database.

screenshot-area-2016-07-18-205651.png

Now on to retrieving this data on the server.

Our backend runs on a python script, so we have a library known as python-firebase which helps us easily fetch the data stored in the Firebase database.

The code for it goes like this

The data will be returned in JSON format, so you can manipulate and store it as you wish.

Well, that’s it!

You now know how to store and retrieve data to and from Firebase.
It makes the work a lot simpler as there is no Database schema or tables that need to be defined, firebase handles this on its own.

I hope that you found this tutorial helpful, and if you have any doubts regarding this feel free to comment down below, I would love to help you out.

Cheers.

Uploading json assets and icons to your app via the app generator

If you have tried out our app generator webpage, you should’ve noticed an option that allows you to upload a zip file which will contain the json for the event.

Why do we need this?

Well, this is needed because not every event organizer can maintain a server and API endpoints which contain the details for their event, so they can simply generate a json for the event by exporting it through the options provided to them on Google Spreadsheets and then they can upload them on the server, so that these files can be packaged in the android and web apps.

Implementation :

The implementation is pretty straightforward and consists of 3 parts :

  1. Making changes to the html file to allow user to upload the zip
<tr>
  <td valign="top">
    <label for="sessions">Zip containing .json files</label>
  </td>
  <td valign="top">
    <input accept=".zip" type="file" id="uploadZip" name="sessions">
  </td>
</tr>

2. Retrieve this file in the javascript and then make an AJAX call to the server

var file_data = $('#uploadZip').prop('files')[0];
 var form_data = new FormData();                 
 form_data.append('file', file_data);
 $.ajax({
            
                  url: '/upload.php', // point to server-side PHP script
                  cache: false,
                  contentType: false,
                  processData: false,
                  data: form_data,                         
                  type: 'post',
                  success: function(php_script_response){
                    // do something
                  }
                });

So here, the form_data contains the details about the file to be uploaded.

In the AJAX call, upload.php takes reads form_data variable and then initiates the upload to the server.

3. Setup a PHP script on the server to respond to the above AJAX call

<?php
    if ( 0 < $_FILES['file']['error'] ) {
        echo 'Error: ' . $_FILES['file']['error'] . '<br>';
    }
    else {
        move_uploaded_file($_FILES['file']['tmp_name'], "/var/www/html/uploads/upload.zip"); 
    }
?>

Here in the PHP script, the file is read and uploaded to a temporary directory in the server.

We then manually copy it to a location and name of our choice.

In case there are multiple users accessing the website and uploading their assets at the same time, we need to pass a timestamp variable to the AJAX call to and later on use it while renaming the uploaded file.

This is to ensure that the file uploaded by one user is not overwritten by another user.

How do we use this data during app compilation

The uploaded zip is then uncompressed and its contents are moved to the assets folder of the android app’s directory.

zip_ref = zipfile.ZipFile(path_to_zip_file, 'r')
zip_ref.extractall(directory)
zip_ref.close()
#TODO: Change path here
for f in os.listdir(directory+ "/zip"):
	if f.endswith('.json'):
		copyfile(f, directoy + "open-event-android/android/app/src/main/assets/"+f)
	elif f.endswith('.png'):
		copyfile(f, directory + "open-event-android/android/app/src/main/res/drawable"+f)
replace(directory+"/open-event-android/android/app/src/main/res/values/strings.xml", 'mipmap/ic_launcher', 'drawable/' + f)

Here replace is a function that searches in the source file for the phrase supplied as it’s argument and changes it with the new phrase.

Now when the app is compiled and ran on a user’s device, it will first search for a json file in the assets directory and if it exists, use that for fetching the data instead of making a network call. For this I have used Gson to first parse the offline files otherwise retrofit makes request to the api and fetches the data from there.

But if there is no json in the assets folder, a normal network call using retrofit will be made and the data will be fetched from the API defined by the user.

Set up Firebase to upload user files

If you’ve read my previous post on uploading files to server, you might have noticed that it was not an easy task to achieve.

There is way too much boilerplate code for uploading a single file, and it will be much more complex in case we plan to upload multiple files, which I need to do for my project.

So what’s the solution to this?

ENTER FIREBASE!

Yeah, you read it right, Firebase once again to the rescue!

I came to know that firebase allows user to store and access the files easily to the built in storage.

Enough chatter for now, lets get to code!

Step 1 :

Add your project to Firebase from the console.

newProj.PNG

Click on the Blue button

Step 2 :

Add Firebase to your webapp

Open the project, you’ve just created and click on the bright red button that says, “ Add Firebase to your web app”

addFirebase.PNGCopy the contents from here and paste it after your HTML code.

Step 3 :

Open the “Storage” tab from the navigation drawer and navigate to the rules tab over there.
We need to set up specific rules as to who all can upload and read files to the storage bucket.

storageRules.PNG
For testing purposes, I’ve allowed everyone to read and write to my storage, but that shouldn’t be the case for your production app

Step 4 :

Add code for uploading your files.

First create a document selection widget and an upload button in your website’s index.html.

<tr>
<td valign=”top”>
<label for=”icon”>Zip File</label>
</td>
<td valign=”top”>
<input accept=”.zip” type=”file” id=”uploadZip” name=”icon”>
</td>
</tr>

Next, create a button to initiate the upload

<tr>
<td colspan=”5″ style=”text-align:center”>
<button type=”submit”>Upload Zip</button>
</td>
</tr>

Next up, inside the JavaScript, add a submitlistener for the submit button and call preventDefault inside it to prevent the form from doing the default action.

var form = document.querySelector(“form”);
form.addEventListener(“submit”, function(event) {
event.preventDefault();

Next up, get a reference to the upload location from your firebase storage bucket.

var timestamp = Number(new Date());
var storageRef = firebase.storage().ref(timestamp.toString());

Next, get the upload button from its ID and add its contents to a variable named file_data.

var $ = jQuery;
var file_data = $(‘#uploadZip’).prop(‘files’)[0];

Now upload that file to firebase.

storageRef.put(file_data);

If everything went as expected, you’ll be able to see the uploaded files onto your firebase console.

 storage

So, you can really appreciate the awesomeness of Firebase by now.
It has replaced the work done by over 50+ lines of code (spread around AJAX calls, PHP Scripts and JavaScript methods) by a single method call.

I would urge you to go through the documentation for more clarity on this.
https://firebase.google.com/docs/storage/

Well, that was it for now.
Next time, I’ll tell you how to retrieve the files back from the storage and add user’s details to Firebase Database.(Yeah, no need for Tables and SQL anymore!)

Cheers. 😀

Sending e-mail from linux terminal

So while finalizing the apk-generator for my GSoC project, I faced a roadblock in sending the generated App to the organizer.

Normally the build takes around 10–12 minutes, so asking the user to wait for that long on the website and then providing him/her with a download link did not feel like a good option. (amirite?)

So I and Manan Wason thought of a different approach to this problem, which was to email the generated app to the organzier.

For doing this, we used 2 handy tools MSMTP and Mutt.
We can use MSMTP to send email but unfortunately we cannot include attachments, so we used Mutt to help us send email with attachments from the command line.
So hang tight and follow the rest of the guide to start sending emails from your terminal and get yourself some developer #swag

Step 1 : Installation

Use the following commands to install MSMTP and Mutt

sudo apt-get -y install msmtp
sudo apt-get -y install ca-certificates
sudo apt-get -y install mutt

We need to have a file that contains Certificate Authority (CA) certificates so that we can connect using SSL / TLS to the email server.

Step 2 : Configuring MSMTP

Now we’ll MSMTP configuration on /etc/msmtprc with the content below. NOTE : You will have to enter your username and password on this file so make sure to make this file private.

Lets first open this file

nano /etc/msmtprc

Next, add following text to the file,

account default
tls on
tls_starttls off
tls_certcheck off
auth on
host smtp.mail.yahoo.com (change this to smtp.gmail.com for gmail)
user username
password password
from [email protected]
logfile /var/log/msmtp.log

NOTE : Refrain from using gmail as they might terminate your account for sending email via MSMTP.

For configuring mutt, we’ll use a similar command to edit the file located at /root/.muttrc

nano /root/.muttrc

Add following text to it which specifies the MSMTP account to use for sending email

set sendmail=”/usr/bin/msmtp”
set use_from=yes
set realname=”MY Real Name”
set [email protected]
set envelope_from=yes

That’s it!
Now lets get ready for the fun part, SENDING THOSE EMAILS 😀

Step 3 : Sending

Now, there are 2 cases that might arise while sending the email,

1 : Sending without attachment

This is pretty straightforward and can be done with either MSMTP or Mutt.

Using MSMTP

printf “To: [email protected]: [email protected]: Testing MSMTPnnHello there. This is email test from MSMTP.” | msmtp [email protected]

Entering the following code will send the email to the recipient and also display the sent email in the terminal.

Using Mutt

mutt -s “Testing [email protected] < /path/to/body.txt

NOTE : ‘body.txt’ is the file whose contents will be used in the body of the email that will be sent to [email protected]

2 : Sending WITH an attachment

Unlike the previous case, this can be done ONLY using Mutt and the code used is

mutt -a /path/to/attachment.txt -s “Testing [email protected] < /path/to/body.txt

The syntax is similar to the above case where we sent the email without attachments.

So well, that it then!
If you followed the instructions carefully, you will have a working email client built into your terminal!
Pretty cool right?

So that’s it for this week, hope to catch you next week with some more interesting tips and tutorials.

Webapp: The generator for making schedule pages

Why a static HTML generator

 

Before we start working on more advanced features like push notifications and iCal exports etc, we have been working on getting a generator (a simple node.js script) up, that can take data for an event either in form of json files, or from a given API in open-event data format, and generate a schedule page.

It has been used to generate the programm page of OpenTechSummit 2016 (http://2016.opentechsummit.net/programm/)

It is based on the open-event-scraper   project of FOSSASIA, and some more features had been added when developing it for OTS16. Some of the new features include –

  • Ability to define copyright and license in the API/Json, and generator adds it in the footer
  • Ability to define sponsors (support for upto 3 levels are there), and the generator adds sponsor logos with links at the bottom of the page.
  • Ability to embed audio, slides and videos into the session cards.

How the process works

Right now if you take a look at the open-event-scraper project under opentechsummit (which is a fork from the FOSSASIA repo, and being used to develop the new features) , you’ll see the process goes like this –

  1. The scraper.py file scrapes data about sessions and speakers from an internal Google Sheet we have. Then the event.py file gets data about the event itself (copyright, links to social channels).

    This step is not needed if we are using a JSON API endpoint. This is needed only if the data source is a Google Sheet, then local JSON files are created.

  2. Once we have an endpoint or JSON files locally downloaded, there a node.js script – generator.js   which generates a static HTML page.
  3. The generated HTML page is based on a handelbars template – schedule.tpl where all the required markup is there.
  4. And finally there is our own stylesheet which is called schedule.css  and is a very lightweight styling addition, on top of what is majorly a vanilla bootstrap layout.

The road ahead

Going forward we will pull back in this source code to our main repo https://github.com/fossasia/open-event-webapp .

Then we’ll add some parameters that can be fed to generator.js when calling it, like  –

  • Name of event
  • Color scheme
  • URL of endpoints

This will have a minimal form-like frontend

We can host this on heroku then, where filling the form, will run the generator.js and the generated HTML and associated CSS files will be available as a zip.