GSoC

AngularJS makes coding CommonsNet easy

Wizard form

Have you seen a wizard form? I am sure you have.  I think the usage of it is very common these days. I like it very much, too. I think its structure is extremely user-friendly, because it can help us to avoid discouraging users from filling the form with all data. As a user I don’t want to be pushed to fill long forms. It’s annoying. But while you use the wizard no matter how long the form is, firstly you can quickly see the progress of you work and then,  don’t see it at once, just fill one field at a time.

That’s the result why have I decided to use it in CommonsNet project, too.

CommonsNet wizard

Angular JS

To implement this on CommonsNet website I have decided to use AngularJS which makes it easy.

I have used Model View Controller which  is a software design pattern for developing web applications. A Model View Controller pattern is made up of the following three parts

Model − It is the lowest level of the pattern responsible for maintaining data

View − It is responsible for displaying all or a portion of the data to the user

Controller − It is a software Code that controls the interactions between the Model and View.

This model helps us to isolate the application logic from the user interface layer and supports separation of concerns.

The ng-controller directive defines the application controller. A controller is a JavaScript Object

Steps to create AngularJS app

1. Load AngularJS on your webiste.

 src = "http://ajax.googleapis.com/ajax/libs/angularjs/1.3.14/angular.min.js">

2. Define AngularJS app using ng-app.

<body ng-app = "app">
   ...
</body>

3. Then I have created views – five different html files for each wizard step. Please take a look at partials folder where you can find them. partials.png

4. Next, I have created a .Controller –  WizardController in AngularJS.

var WizardController = app.controller("WizardController", function ($scope) {
// controller logic goes here
}
);
wizard controller.png

 

It is a JavaScript function. I have defined which  view has to be displayed by .controller on each step. As you can see it’s quite easy and clear. You can quickly define and see which step is it , how is its name and which template should be used. It definitely enables you to maintain your app easier, because changes can be made quickly and does not influence on other part of code so you can take control over your code.

5. Then I have used  this line of code to call to my WizardController to display and maintain my views and that’s it.  See the progress on CommonsNet

<div id="wizard-container" ng-controller="WizardController as vm">
Continue ReadingAngularJS makes coding CommonsNet easy

Writing vim scripts to open files in the steam structure

My objective was to write a script that will allow the user to open new files from inside the vim interface. There is a vim command available for normal files. However here I am not talking about normal files, these are files inside the steam structures, that is , inside rooms and containers. The commands that were doing this currently were edit.pike and the edit command from the steam-shell.

Issue opened: https://github.com/societyserver/sTeam/issues/53

As I started working on it one of the issue I faced was that I could not use the code in edit.pike or the edit command in steam-shell because for that I would have had to start a new process and the vim window would have come up as an another new process. Due to these process using the same terminal window there would be an overlap and vim will not be able to function, this was one of the problems with the original implementation of the edit command, which I had solved in the first week of GsoC.

Issue with edit command: https://github.com/societyserver/sTeam/issues/34
Solution: https://github.com/societyserver/sTeam/pull/36

My colleague Ajinkya Wavare had finished his task wherein we could execute pike code from a vim terminal. For this he had modified steam-shell and was passing the pike code as an argument to the call of steam-shell. One advantage of executing pike code this way is that all the steam objects and variables are available to use with the pike code. Basically it is like executing code on debug.pike. I based my solution to the problem on this newly added feature. I was able to pass in pike code to steam-shell, this would start a new process, execute the code and return me the output.

The tasks that I needed to do with the pike code was:

  • Find the required object in the steam server.
  • Get the content of the object.
  • Save the content in a temporary file.
  • Once the file is saved by vim, update the file on the sever and the logs in the vim buffer.

I created a new steam command ‘Open’ and passed the full path of the object as an argument. Given the full path it was easy to find the object on the sever. I was able to fetch the contents and save it in a file. Now I had a big problem. The name of the temporary file was available in the pike script, I needed this name in the vim script to open the file in a new tab. I spend more than a day trying to come up with a solution for this problem. I was able to get the name of the file in the output of the pike script and this output could be read into a vim buffer. However this output had a lot of other content including the result of execution of other pike statements also. I had to use vim search and select tools to get the file name isolated and stored in a vim script variable. Once I achieved this my task was almost done. I used vim command to open this file and the log in a new vim tab.

Result of execution of pike script containing the file name.
Result of execution of pike script containing the file name.

The next step was to get this file to be uploaded to the server and the logs updated. Ideally when a file is opened from the steam-shell using the edit command, there is a piece of code that gets called every one second and performs this tasks, however this couldn’t be used when the file is opened from inside vim as the steam-shell process that gets the file closes before the file is opened in vim. So I couldn’t have a pike script constantly watching the file. The solution I came up with was using auto commands to execute a pike script when the file is saved and update the file on the server and also the logs. This completed my task.

opening from vim
File being opened from inside vim.

 

file opened from vim
File opened in new tab.

Since I had a day left in the week I took up one more small task. This was due to a problem that arose due to my previous task that is letting users open multiple files for editing. There were too many vim buffers open at a time and it was a trouble closing them as :q used to be executed for each buffer. Therefore I made a used defined command in vim, :Q, that was able to close tabs at a time. :tabclose is a vim command that does the same thing however it cannot close the last tab, this shortcoming was overcome with :Q.

Issue: https://github.com/societyserver/sTeam/issues/62

solution: https://github.com/societyserver/sTeam/pull/65

Continue ReadingWriting vim scripts to open files in the steam structure

Migrations make us crazy!

Our Open Event team is not small, so problems with migrations occur very often. I’d like to describe how to solve these common problems and avoid Contributors frustration. Because more of us didn’t know how to solve this problems at the beginning so we wasted a lot of time to find a source of problem.

The most common mistake is that we forget run migration on Heroku. Developer is sometimes surprised because something works for him but he forgets to run migration on server. These small mistakes lead to huge problems and at that time our app throws a lots of bugs related to database. We can often see “Internal server error”.Screen Shot 2016-06-17 at 22.55.25.png So if developer changes table he has to run migration!

vagrant@vagrant-ubuntu-trusty-64:/vagrant$ python manage.py db migrate

But above command Quite often doesn’t solve our problem, because We face other problems while updating DB for example

alembic.util.exc.CommandError: Multiple head revisions are present for given argument ‘head’; please specify a specific target revision, ‘<branchname>@head’ to narrow to a specific head, or ‘heads’ for all heads

this problem is caused by two developers which push code to Github with migrations without merging two heads to achieve one head.

So to solve this problem you only have to know ids of two heads

vagrant@vagrant-ubuntu-trusty-64:/vagrant$ python manage.py db heads

e38935822969 (head)
f55fde3d62b1 (head)

Then you have to merge it

vagrant@vagrant-ubuntu-trusty-64:/vagrant$ python manage.py db merge e38935822969 f55fde3d62b1

Generating /vagrant/migrations/versions/ecb671d1eb4b_.py … done

Upgrade DB

vagrant@vagrant-ubuntu-trusty-64:/vagrant$ python manage.py db upgrade

INFO [alembic.runtime.migration] Context impl PostgresqlImpl.
INFO [alembic.runtime.migration] Will assume transactional DDL.
INFO [alembic.runtime.migration] Running upgrade 00ea66754d06 -> d84f976530e1, empty message
INFO [alembic.runtime.migration] Running upgrade d84f976530e1 -> 1b3e4f5f56bd, empty message
INFO [alembic.runtime.migration] Running upgrade 1b3e4f5f56bd -> e38935822969, empty message
INFO [alembic.runtime.migration] Running upgrade e38935822969, f55fde3d62b1 -> ecb671d1eb4b, empty message

And finally run migrations

vagrant@vagrant-ubuntu-trusty-64:/vagrant$ python manage.py db migrate
INFO [alembic.runtime.migration] Context impl PostgresqlImpl.
INFO [alembic.runtime.migration] Will assume transactional DDL.
INFO [alembic.ddl.postgresql] Detected sequence named ‘role_id_seq’ as owned by integer column ‘role(id)’, assuming SERIAL and omitting
INFO [alembic.ddl.postgresql] Detected sequence named ‘microlocation_id_seq’ as owned by integer column ‘microlocation(id)’, assuming SERIAL and omitting….

Continue ReadingMigrations make us crazy!

Increasing utility of sTeam tools

(ˢᵒᶜⁱᵉᵗʸserver) aims to be a platform for developing collaborative applications.
sTeam server project repository: sTeam.

User Utils

User  convenience is an important aspect for any application to succeed. The sTeam root user should be able to create or delete a user. A command to add / delete a user was added successfully.

The root user should be able to add new user’s using the command line.
The parameter’s like username, password, email-id etc. should be asked and then the user should be created.

Issue. Github Issue Github PR
Create a user. Issue-58 PR-59
Delete an object Issue-56 PR-57
Delete a user. Issue-69 PR-70

To create a new user:

create_user username password email

The current users can be found by running the command

 _Server->get_module("users")->get_users();

Similarly a command to delete the user was added to the steam-shell.

The user should be able to delete the objects created/existing inside the user area. A steam-shell command needs to be added to delete the objects from the command line.

The usage of command to delete a user inside steam-shell.pike. Only the root user can delete other user.
Command:

delete_user username

UserActivities
In order to delete an object which can be a container, document or a room in the current steam directory, run the command;

delete test.pike

The object would be deleted.

Tab-Completion Module

The tab completion module of the sTeam shell was analyzed during this period. The tab completion module has an issue whereby it doesn’t lists the options on pressing the tab after  ".

query_attribute("

After pressing the tab after ” the options should be listed. But this is not the case. The bug needs to be resolved.

Tab_completion

Import from git script

The import to git script can now import a single file into the sTeam directory.

The feature for this was added.

Issue. Github Issue Github PR
Add utility to support single import in import-from-git script . Issue-16 PR-76 
sudo ./import-from-git.pike gitfolder/xyz.mp3 /home/sTeam/

The xyz.mp3 would be imported to the sTeam directory.

Import-script

The future work would include resolving the tab_completion module issues and enhancing the import script to support the feature where by a user would be able to specify the name of the file in the sTeam directory.

Checkout the FOSSASIA Idea’s page for more information on projects supported by FOSSASIA.

Continue ReadingIncreasing utility of sTeam tools

Sending e-mail from linux terminal

So while finalizing the apk-generator for my GSoC project, I faced a roadblock in sending the generated App to the organizer.

Normally the build takes around 10–12 minutes, so asking the user to wait for that long on the website and then providing him/her with a download link did not feel like a good option. (amirite?)

So I and Manan Wason thought of a different approach to this problem, which was to email the generated app to the organzier.

For doing this, we used 2 handy tools MSMTP and Mutt.
We can use MSMTP to send email but unfortunately we cannot include attachments, so we used Mutt to help us send email with attachments from the command line.
So hang tight and follow the rest of the guide to start sending emails from your terminal and get yourself some developer #swag

Step 1 : Installation

Use the following commands to install MSMTP and Mutt

sudo apt-get -y install msmtp
sudo apt-get -y install ca-certificates
sudo apt-get -y install mutt

We need to have a file that contains Certificate Authority (CA) certificates so that we can connect using SSL / TLS to the email server.

Step 2 : Configuring MSMTP

Now we’ll MSMTP configuration on /etc/msmtprc with the content below. NOTE : You will have to enter your username and password on this file so make sure to make this file private.

Lets first open this file

nano /etc/msmtprc

Next, add following text to the file,

account default
tls on
tls_starttls off
tls_certcheck off
auth on
host smtp.mail.yahoo.com (change this to smtp.gmail.com for gmail)
user username
password password
from username@yahoo.com
logfile /var/log/msmtp.log

NOTE : Refrain from using gmail as they might terminate your account for sending email via MSMTP.

For configuring mutt, we’ll use a similar command to edit the file located at /root/.muttrc

nano /root/.muttrc

Add following text to it which specifies the MSMTP account to use for sending email

set sendmail=”/usr/bin/msmtp”
set use_from=yes
set realname=”MY Real Name”
set from=username@yahoo.com
set envelope_from=yes

That’s it!
Now lets get ready for the fun part, SENDING THOSE EMAILS 😀

Step 3 : Sending

Now, there are 2 cases that might arise while sending the email,

1 : Sending without attachment

This is pretty straightforward and can be done with either MSMTP or Mutt.

Using MSMTP

printf “To: recipient@domain.comnFrom: username@domain.comnSubject: Testing MSMTPnnHello there. This is email test from MSMTP.” | msmtp recipient@domain.com

Entering the following code will send the email to the recipient and also display the sent email in the terminal.

Using Mutt

mutt -s “Testing Mutt” — recipient@domain.com < /path/to/body.txt

NOTE : ‘body.txt’ is the file whose contents will be used in the body of the email that will be sent to ‘recipient@domain.com’.

2 : Sending WITH an attachment

Unlike the previous case, this can be done ONLY using Mutt and the code used is

mutt -a /path/to/attachment.txt -s “Testing Mutt — recipient@domain.com < /path/to/body.txt

The syntax is similar to the above case where we sent the email without attachments.

So well, that it then!
If you followed the instructions carefully, you will have a working email client built into your terminal!
Pretty cool right?

So that’s it for this week, hope to catch you next week with some more interesting tips and tutorials.

Continue ReadingSending e-mail from linux terminal

Implementing TLS in pike

My next task was an enhancement. The command line client for sTeam was not using any kind of encryption and all the data sent over a network was in clear text. Now this was a little different from normal tasks because,

1. TLS had to be implemented in Pike.

2. TLS was to be implemented over COAL.

COAL is a home grown protocol specially for sTeam. My mentor Martin helped me to go about this task. He taught me how to break a task into small ones. I began with writing an SSL client in pike that can interact with an https server. It turned out pike has an SSL module that just made the task very simple. I had to understand how SSL works however pike made the task quite easy. I also received constant help from the pike mailing list where people actively guided me in the right direction.

Once I was done with the SSL client I approached the actual problem. I had to understand the functioning of COAL and narrow down the particular files involved. I had to wrap the COAL protocol in TLS so that it becomes COALS. Pike allows users to import programs at objects. This was used to import client_base.pike. This file involved the code for the connection. I had to throughly go through this and the files that it imported that is kernel/socket. After several experiment with these files I was able to use the SSL module and successfully make the connection use the TLS protocol. I checked this using wireshark, which captured all the packets over the network and all the data could be seen as encrypted.

wireshark clear text
clear text data over network
wireshark ssl
encrypted data after implementing TLS protocol
Continue ReadingImplementing TLS in pike

Open Event Apk generator

So we made this apk generator currently hosted on a server (http://192.241.232.231) which let’s you generate an android app for your event in 10 minutes out of which the server takes about 8 minutes to build 😛 . So, essentially you just have to spare 2 minutes and just enter 3 things(email, Desired app’s name and Api link). Isn’t this cool?

So how exactly do we do this?

At the backend, we are running a python scripts with some shell scripts where the python script is basically creating directories, interacting with our firebase database to get the data entered by a user. So we made these scripts to first of all to clone the open event android repo, then customise and change the code in the repo according to the parameters entered by the organiser earlier(shown in the image).

Screen Shot 2016-06-14 at 12.13.12 AM
Generator Website

After the code has been changed by the scripts to customise the app according to the event the app will be used for, we move on to the script to build the apk, where we build and sign the apk in release mode using signing configs and release keys from the server because we don’t the organiser to generate keys and store it on the server to avoid the hassle and also the privacy concerns involving the keys. So this is when the apk is generated on the server. Now you have questions like the apk is generated but how do I get it? Do I have to wait on the same page for 10 minutes while the apk is being sent? The answer is no and yes. What I mean by this is that you can wait to download the apk if you want but we’ll anyways send it to your email you specified on the apk generator website. This is the sample Email we got when we were testing the system

Screen Shot 2016-06-14 at 12.08.59 AM.png

So it’s an end to end complete solution from which you can easily get an android app for your event in just 2 minutes. Would love to hear feedback and reviews. Please feel free to contact me @ manan13056@iiitd.ac.in or on social media’s(Twitter, Facebook). Adios!

P.S. : Here is the link to the scripts we’re using.

Continue ReadingOpen Event Apk generator

Why should you use Heroku?

Last week I’ve dedicated most time to implement new heroku features to Open Event. Due to the fact that wasn’t easy tasks I can share my experience.

What is heroku?

Heroku is a cloud Platform-as-a-Service (PaaS) supporting several programming languages like Java, Node.js, Scala, Clojure, Python, PHP, and Go

Easy to deploy

what you need to do:

  1. Create account on heroku
  2. Download a heroku toolbelt to your enviroment.
  3. Go to your project directory(/open-event-orga-server)
  4. Sign in to heroku in your command line using your credentials 
    $ heroku login
  5. Create app with name $ 
    heroku apps:create your_app_name

    after execution above command you will recive a link to your application

    http://your_app_name.herokuapp.com/
  6. Push latest changes to heroku 
    $ git push heroku master
  7. If everythings is ok you can check results http://your_app_name.herokuapp.com/ (sometimes it does not work like we want 🙁 )

Easy to configure

To list/set/get config variables use:

$ heroku config
$ heroku config:set YOUR_VARIABLE=token123

$ heroku config:get YOUR_VARIABLE

or you can go to you application dashboard and make above operations

How you can get access to this variables using python langauges?

$ python

Python 2.7.10 (default, Oct 23 2015, 19:19:21) 

[GCC 4.2.1 Compatible Apple LLVM 7.0.0 (clang-700.0.59.5)] on darwin

Type "help", "copyright", "credits" or "license" for more information.

>>> import os

>>> your_variable  = os.environ.get('YOUR_VARIABLE', None)


>>> print your_variable
token123

I’ve used this to display current release in my python application(You need to generate a special API token and add it to config variables)

os.popen('curl -n https://api.heroku.com/apps/open-event/releases -H 
"Authorization: Bearer ' + token + '" -H 
"Accept: application/vnd.heroku+json; version=3"').read()

Easy to monitor

If something is wrong with your APP you need to use this command

$ heroku logs

it shows all logs

To see 10 latest releases use:

$ heroku releases

How you can set up Open Event to deploy to heroku?

  1. Clone https://github.com/fossasia/open-event-orga-server
  2. Go to directory of open event orga server(/open-event-orga-server)
  3. Add git remote 
     heroku git:remote -a open-event
  4. You can check if open event is added to git remote 
    $ git remote -v
heroku https://git.heroku.com/open-event.git (fetch)
heroku https://git.heroku.com/open-event.git (push)
origin https://github.com/fossasia/open-event-orga-server.git (fetch)
origin https://github.com/fossasia/open-event-orga-server.git (push)
  5. Now you can deploy changes to open-event application(You need a permissions 🙂 )

Why should you use a Heroku?

It’s great to deploy apps because you are able to share content in short time what I’ve done. Besides it’s very well documented so you can find there answers for most of your questions. Finally most of things you can configure using Heroku dashboard so it’s the best advantages of this tool.

Continue ReadingWhy should you use Heroku?

Improvements in sTeam shell and export script

(ˢᵒᶜⁱᵉᵗʸserver) aims to be a platform for developing collaborative applications.
sTeam server project repository: sTeam.

Logs

A breakthrough in sTeam development took place. Siddhant Gupta was able to successfully implement the TLS protocol in pike.

The Logs were displayed erroneously. Also the user was not able to scroll down the log to the latest message.  The golden ratio script used in the program was updated to it’s latest version.

Also the editor is opened using sudo command so as to access the vim scrips in the /usr/local/lib/steam/tools directory.
The files are opened using the vim command:
vim -S script -c edit filename1|sp filename2

Only one buffer is accessible at a time. In order to switch the buffer to log buffer the command is CTRL+Ww. Enter this command directly without entering the vim terminal using :.
The log buffer would be accessible, can be edited and scrolled down to the latest log message.

Issue. Github Issue Github PR
Access the log window till the end. Issue-20 PR-48
Open appropriate log window when a sTeam command is executed. Issue-49 PR-51

ScrollToLatest

The logs were displayed erroneously. Ideally the log should be displayed based on the buffer where the sTeam function is called and accordingly the relevant log buffer should be called and display the output. This error was fixed.

MultTabs

The log is displayed in the file named after the file which is opened and concatenated with the suffix “-disp”. In Vi the  :% buffer stores the value of the current file. Consequently this value was concatenated with “-disp” to display the buffer accordingly. The Vi script can be seen below.
ViFunc

Export to git script.

The  export-to-git script was tested vigorously.  All the known issue’s based on the script were replicated in the system and solution was found for them.

The export to git script is now capable of exporting multiple sources at a time. If the last argument is always the target repository then any number of previous arguments can be sources.

Issue. Github Issue Github PR
Support Multiple Source arguments Issue-14 Issue-19 PR-54
Include Source-name in branch name and add branch description. Issue-9 PR-55

Example command :
./export-to-git.pike /home/sTeam/file1 /home/coder/file2 /home/sTeam/container3 ~/gitfolder

The export-to-git script also exports the source name in the branch.  To help distinguish between the branches, we need more descriptive names:

./export-to-git.pike /sources/ /tmp/export-test/

This should create the branch sources-cur_time. Also when a file is specifically exported:

./ecport-to-git.pike /home/coder/demo1.txt /temp/export-test/

would create a branch with name home/coder/demo1.txt-cur_time. This is done to avoid ambiguity between files with same name existing in different locations.
Also a description is added fo the branch name using the git command
git config branch.<branch name>.description "describe branch"
To view this description go to the folder where the branch is exported and then enter the git command
git config branch.<branch name>.description

Export to Git script executing when Multiple Source arguments passed and the modified branch name.
GitExportAndBranchName
Branch Description

BranchDesc

Checkout the FOSSASIA Idea’s page for more information on projects supported by FOSSASIA.

Continue ReadingImprovements in sTeam shell and export script

steam-shell: Two processes in one terminal

Community bonding period turned out to be quite fruitful I got to know my community really well and not only that I also solved quite a number of issues which helped me understand the code base. Daily scrum meetings played a very important role in making us work professionally and cover some substantial work. Official coding period began on 23rd May and I was all set for the challenges and the sleepless nights to come. Here I will be discussing the tasks I covered in my first week.

As suggested by my mentors I had changed my plans a bit by moving the work on edit command before implementing the TLS layer on COAL. I started small by fixing the edit command. The edit command opens the specified file in vi/vim/emac. In vi and vim the editor was misbehaving and not letting us work on the file. I took up this as my first task for Google Summer of Code 2016. This helped me understand steam-shell and applauncher, which is used to load the editor, in detail. Vi and vim editors have an advantage of letting the user edit the file in the same terminal window.

Looking at the issue itself it was not possible to do any kind of backtracking. The vi editor was just throwing rubbish on the screen when the user attempted to type anything.

overlapping process
vi editor showing the garbage and the steam-shell command

At first I was under the impression that it was a problem with the editor itself. I even tried approaching the vi.stackexchange.com , where the vi developers could help me. However all this was in vain. After a lot more forensics and re-reading the code multiple number of times I realized two process were active and sharing the same terminal space. How did I come to this conclusion? Well it was a very minute detail that I noticed. While in the vi editor window, with the document open and the editor throwing garbage at you when you press the up arrow the editor clears some area and show the commands executed on the steam-shell. This can be seen in the above image

This simply meant that both the process for steam-shell and the vi editor was running and sharing the same terminal space. The solution was quite simple. Just called editor→wait() to suspend the calling the process till the called process was over.

Continue Readingsteam-shell: Two processes in one terminal