sTeam GSoC 2016 Windup

(ˢᵒᶜⁱᵉᵗʸserver) aims to be a platform for developing collaborative applications.
sTeam server project repository: sTeam.
sTeam-REST API repository: sTeam-REST

An overview of the work done by ajinkya007 during Google Summer of code 2016 with FOSSASIA on its project sTeam.

The community bonding period saw the creation of a docker image and a debian package for the sTeam server. The integration of the sTeam shell into vi, improvements in the export and import to git scripts, user and group manipulation commands, sending mails through the commandline, viewing logs and the edit script modifications were done subsequently. The later part of GSOC saw that the sTeam-rest repository was restructured, unit and api-end point tests were performed. The new web interface developed was tested.
The code written during this period by me and siddhant was merged and the conflicts were resolved. The merged code was tested thoroughly as no automated test integration tool supports pike programming language. Documentation was generated using Doxygen and deployed in the gh-pages of the sTeam server repository.

A trello board was maintained throughout the course of GSOC 2016.

Trello Board: sTeam

Accomplishments

Issues Reported and Resolved

A list of tasks covered and all the Pull requests related to each:

Tasks Issue PR
Make changes in the Makefile for installation of sTeam. Issue-25 Issue-27 PR-66 PR-67
Edit script modifications Issue-20 Issue-29 Issue-43 PR-44 PR-48
Indentation of output in steal-shell. Issue-24 PR-42
Integrate steam-shell into vim or emacs. Issue-37 Issue-43 Issue-49 PR-41 PR-48 PR-51
Improve the import and export from git scripts. Issue-9 Issue-14 Issue-16 Issue-18 Issue-19 Issue-46 PR-45 PR-54 PR-55 PR-76
Create, Delete and List the user through commandline Issue-58 Issue-69 Issue-72 PR-59 PR-70 PR-78
Sending Mails through commandline Issue-74 PR-85
Generate error logs and display them in CLI Issue-83 PR-86
Create a file of any mime type from command line. Issue-79 PR-82
Add more commands for group operations. Issue-80 PR-84
Add more utility to the steam-shell Issue-56 Issue-71 Issue-73 PR-57 PR-75 PR-81
Restructure the sTeam-rest repository List of Issue’s List of PR’s
Write test cases to test sTeam-rest api List of Issue’s List of PR’s
Create a debian package and a docker image for easy deployment Create docker image Docker Image
Document the work done Issue 149 sTeam Server Structure, sTeam Server Documentation
Test the web-interface

Commits Merged

During the course of GSOC 2016, work was done on the sTeam and sTeam-rest repositories.

1. The work done on the sTeam repository.

We have combined all the work into two branches for the ease of creating a debian package. The commits made by me in each branch can be seen here.

2. The work done on the sTeam-rest repository

The push request’s sent for the issue’s are yet to be merged in the main repository. The list of PR’s for the sTeam-rest repository.

sTeam-rest PR’s

The weekly blogs

The blogs summarizing the work done during the week were published on my personal website. These can be found on Weekly Blogs
All the blogs can also be found on the Fossasia blog.
The list in reverse chronological order is as follows.

Scrums

Scrum reports were posted on the #steam-devel on irc.freenode.net and sTeam google group. The sTeam trello board also has everyday scrum reports.

Further Improvements

  1. sTeam command line lacks the functionality to read and set the object access permissions. sTeam function analogous to getfacl() to change the sTeam server object permisssions.
  2. sTeam debian package for easy installation of the sTeam server. The debian package is yet to be fully packaged.

Special Thanks

  • I would like to thank my mentors Mario Behling, Hong Phuc Dang, Martin Bahr, Trilok Tourani and my peers for being there to help me and guide me.
  • I would like to thank FOSSASIA, sTeam and Pike Community for giving me this opportunity and guiding me in this endeavour.
  • I would also like to thank Google Summer of Code for this experience.

Feel free to explore the repository. Suggestions for improvements are welcomed.

Checkout the FOSSASIA Idea’s page for more information on projects supported by FOSSASIA.

PIL to convert type and quality of image

Image upload is an important part of the server. The images can be in different formats and after applying certain javascript modifications, they can be changed to different formats. For example, when an image is uploaded after cropping in open event organizer server, it is saved in PNG format. But PNG is more than 5 times larger than JPEG image. So when we upload a 150KB image, the image finally reaching the server is around 1MB which is huge. So we need to decide in the server which image format to select in different cases and how to convert them.

Continue reading PIL to convert type and quality of image

Templating with Handlebars and Nodejs

There are many frameworks present today that does DOM manipulation. But, DOM manipulation is useful only with simpler Javascript apps. If we want to deal with huge amount of data, we will need to take the support of templating.

One of the templating libraries is Handlebar.js. Handlebars.js is the lightest and one of the fastest templating libraries I have worked with.

How to work with Handlebars ?

Handlebars can be said as the superset of Mustache templating language. In Open-event-web app, Handlebars is used along with Node.js.

1. To install Handlebars
npm install handlebars
2. To include handlebars
const handlebars = require('handlebars');
const fs         = require('fs-e'xtra);
3. To compile a file ( Extension .tpl or hbs)
const tpl = handlebars.compile(fs.readFileSync(__dirname +   '/templates/schedule.hbs').toString('utf-8'));
fs.writeFileSync(distHelper.distPath + '/' + appFolder + '/tracks.html', tpl(jsonData));

Here jsonData is the JSON Array of objects provided to the template.

An example of a .hbs templating file can be :

 <div class="track-names col-md-2">
 {{# tracks}}
 {{#if title}}
 
 <ul class="title-inline">
 <li style="background-color:{{{color}}}" class="titlecolor"></li>
 <li>{{title}}</li>
 </ul>
 
 {{/if}}
 {{/tracks}} 
 </div>

 

 

KISS Datatable

Recenlty I’ve faced a problem with sorting columns in Datatable.

What is Datatable?

Datatable is a plug-in for Jquery library. It provides a lot of features like pagination, quick search or multi-column ordering. Besides, you can develop easily Bootstrap or Foundation ui css styles. There are also more other option but It doesn’t make sense to list it here, because you can visit their site and you can read clearly documentation. On Datatable website you can see a lot of examples. First of them shows how to improve your ordinary table to awesome and rich of features table. One function changes everything, It’s fantastic!  

$('#myTable').DataTable();

Returning to my problem which I’ve faced, as I told it was problem related to sorting column in table.

I know sorting is a trivial thing. I hope that everyone knows it 🙂 Sorting by a date is also implemented in a datatable library. So everything is clear when we don’t change date format to human readable format. I mean something like this ‘3 hours ago’, ‘1 year ago’.

When Open Event team tested how datatable manages ordering columns in that format it didn’t work. It’s quite hard to sort by that format, So I’ve invented an idea. Surely you are wondering what i’ve invented. I’ve postponed my minds about sort by this values. It can direct to overwork. When I thought about it, weird ideas came to my mind, a lots of conditions, If statements… Therefore I’ve resigned from this. I’ve used KISS principle. KISS means ‘keep it simple stupid’. I like it!

Therefore that sorting is implemented on frontend side. I’ve decided not to display human readable date format at the beginning. I find  all dates which have format “YYYY-MM-DD HH:mm” then I replace that format to human readable format. So it’s very quick and comfortable, and doesn’t require a lot conditions to write. Of course I’ve tried to implement it in Datatable library. I suppose that it would  require more effort than it’s now.

Below You can see great function which changes a date in frontend side but does not change data in a datatable. So sorting process takes place in a datatable using format  “YYYY-MM-DD HH:mm” but user see human readable format. Isn’t it awesome?!

function change_column_time_to_humanize_format(datatable_name, column_id) {
  $(datatable_name).each(function( key, value ) {
    $(value).children().each(function( key1, value2 ) {
       if(key1 === column_id ){
          var current_value = $(value2).text().slice(0, 16);
          var changed_value = moment(current_value, "YYYY-MM-DD hh:mm").fromNow()
          var isValid = moment(current_value, "YYYY-MM-DD HH:mm", true).isValid()
          if (changed_value !== current_value && isValid === true){
              $(value2).text(changed_value)
          }
      }
  });
});

 

sTeam Docker Image

(ˢᵒᶜⁱᵉᵗʸserver) aims to be a platform for developing collaborative applications.
sTeam server project repository: sTeam.
sTeam-REST API repository: sTeam-REST

DOCKER IT!

What is Docker?

Docker is open source software to pack, ship and run any application as a lightweight container. Containers are completely hardware and platform independent so you don’t have to worry about whether what you are creating will run everywhere.

In order to facilitate development in all the environments and give the user the ease from the cumbersome installation steps a docker image was made for the sTeam project.
This docker image contains all the necessary dependencies to start the sTeam server.
It also includes the sTeam UI and the Rest API along with it’s dependencies. These have already been installed and developer can start collaborating to it.

docker-image

The docker image can be found at :


https://hub.docker.com/r/ajinkya007/societyserver/

It has all the necessary information for it’s usage and installation.

The dockerfile for the sTeam repository:


FROM ubuntu:latest
RUN apt-get update
RUN apt-get install -y wget git firefox nodejs nodejs-legacy npm
RUN apt-get install -y build-essential mysql-server libmysqld-dev bzip2 libjpeg$
RUN apt-get install -y pike7.8 pike7.8-bzip2 pike7.8-svg
RUN apt-get install -y libxml2-dev libxslt1-dev automake flex
RUN npm install -g npm bower gulp coffee-script jasmine-node frisby
RUN cd home
RUN wget -c https://raw.githubusercontent.com/societyserver/sTeam/steam-package$
RUN git clone www.github.com/societyserver/steam.git
RUN cd sTeam
RUN git checkout societyserver-source
RUN ./build
RUN sudo ./install
RUN cd ..
RUN git clone https://github.com/societyserver/sTeam-web-interface-gsoc-2015.git
RUN cd sTeam-web-interface-gsoc-2015
RUN npm install
RUN cd ..
RUN git clone https://github.com/societyserver/steam-rest.git
RUN cd steam-rest
RUN npm install

Feel free to explore the repository. Suggestions for improvements are welcomed.

Checkout the FOSSASIA Idea’s page for more information on projects supported by FOSSASIA.

Twitter Section Using loklak webtweets

In Open event web app, the user can provide URL of social links such as Twitter, Facebook etc in the event.json file inside the ZIP. The previous functionality was to use Twitter API and to generate a timeline showing the tweets of the twitter URL mentioned in event.json by user. But, it can be done by following another approach which reduces the third party dependency i.e Loklak-webtweets.

I have implemented the twitter section using loklak webtweets which can be done very easily.

Step 1:  Including necessary files from loklakwebtweets repository inside index.html. You can find them in js/ folder of this repository.

<script src="./dependencies/jquery.min.js"></script>
<script src="./dependencies/bootstrap.min.js" type="text/javascript"></script>
<script src="./dependencies/loklak-fetcher.js" type="text/javascript"></script>
 <script src="./dependencies/tweets.js" type="text/javascript"></script>

 

Step 2:  Specify the data source in HTML from which twitter data will be fetched. Here I have extracted the last word from the twitter URL provided by the user and passed it to HTML.

const sociallinks = Array.from(event.social_links);
 var twitter ="";
 sociallinks.forEach((link) => {
  if(link.name.toLowerCase() === "twitter") {
   twitter = link.link;
  }
 }) 
 const arrayTwitterLink = sociallink.split('/');
 const twitterLink = arrayTwitterLink[arrayTwitterLink.length - 1];
 
 const urls= {
   twitterLink: twitterLink,
   tweetUrl: twitter,
 };

This code will search twitter link in social links array present in event.json and get its last character which will be provided to data-from and data-query attribute of HTML.

 <section class="sponsorscont">
  <div class="tweet-row">
   <div class="col-sm-12 col-md-12 col-xs-12">
    <i class ="social_twitter fa fa-twitter"></i>
     <div class="tweets-feed" id="tweets" data-count=50 data-query="    {{{eventurls.twitterLink}}}" data-from="{{{eventurls.twitterLink}}}">
     <div class="arrow-up"></div>
      <p id="tweet" class="tweet">
       Loading...
     </p>
   <span style="margin-bottom: 20px;" id="dateTweeted"></span>
    <p>Follow<u>
    <b><a href="{{eventurls.tweetUrl}}"/>
     @{{eventurls.twitterLink}}</a>
    </b></u> for more updates</p> 
     </div> 
    </div>
  </div>
</section>

Step 3 : Now we just need to add styling so that it looks decent. For that, I have written some SASS.

.tweets-feed {
   color: $black;
   line-height: 30px;
   font-size: 20px;
   transition: opacity 0.2s linear;
   margin-bottom: 20px;
   height: 100px;
 
  a {
   color: $black;
   text-decoration: underline;
   font-weight: 700;
  } 

  #dateTweeted {
   font-size: 15px;
   display: block;
  }

}

.tweet-row {
   padding: 0 80px;
   margin-bottom: 80px;
   .social_twitter {
     font-size: 60px;
     margin-bottom: 12px;
  }
}

The output from the above code is a well designed Twitter section fetching tweets from the URL provided as a string in event.json by user.

tweet

 

R14 – Memory Quota Exceeded

We, like many other organisations, are using heroku as the deployment server for our project open event organizer server. Things are pretty simple and awesome when your project is in its beginning phase and things run pretty smoothly. But as your project grows, there comes some server problem. And one of the biggest problems as your project grows is memory. Now since various packages have a different amount of memory assigned to you in case of hosting in generic servers such as heroku, so it might result in memory quota exceeded. Recently, we faced such a problem. R14 – Memory Quota Exceeded. Took us quite some time to understand what and why and how this occurred. So let me share a few things I found about this error.

Continue reading R14 – Memory Quota Exceeded

Python code examples

I’ve met many weird examples of  behaviour in python language while working on Open Event project. Today I’d like to share some examples with you. I think this knowledge is necessary, if you’d like to increase a  bit your knowledge in python area.

Simple adding one element to python list:

def foo(value, x=[]):
  x.append(value)
  return x

>>> print(foo(1))
>>> print(foo(2))
>>> print(foo(3, []))
>>> print(foo(4))

OUTPUT

[1] 
[1, 2] 
[3]
[1, 2, 4]

First output is obvious, but second not exactly. Let me explain it, It happens because x(empty list) argument is only evaluated once, So on every call foo(), we modify that list, appending a value to it. Finally we have [1,2, 4] output. I recommend to avoid mutable params as default.

Another example:

Do you know which type it is?

>>> print(type([ el for el in range(10)]))
>>> print(type({ el for el in range(10)}))
>>> print(type(( el for el in range(10))))

Again first and second type are obvious <class ‘list’>, <class ‘set’>. You may  think that last one should return type tuple but it returns a generator <class ‘generator’>.

Example:

Do you think that below code returns an exception?

list= [1,2,3,4,5]
>>> print(list [8:])

If you think that above expression returns index error you’re wrong. It returns empty list [].

Example funny boolean operators

>>> 'c' == ('c' or 'b')
True
>>> 'd' == ('a' or 'd')
False
>>> 'c' == ('c' and 'b')
False 
>>> 'd' == ('a' and 'd')
True

You can think that that OR and AND operators are broken.

You have to know how python interpreter behaves while looking for OR and AND operators.

So OR Expression takes the first statement and checks if it is true. If the first statement is true, then Python returns object’s value without checking second value. If first statement is false interpreter checks second value and returns that value.

AND operator checks if first statement is false, the whole statement has to be false. So it returns first value, but if first statement is true it checks second statement and returns second value.

Below i will show you how it works

>>> 'c' == ('c' or 'b')
>>> 'c' == 'c'
True
>>> 'd' == ('a' or 'd')
>>> 'd' == 'a'
False
>>> 'c' == ('c' and 'b')
>>> 'c' == 'b'
False 
>>> 'd' == ('a' and 'd')
>>> 'd' == 'd'
True

I hope that i have explained you how the python interpreter checks OR and AND operators. So know above examples should be more understandable.

sTeam Server Object permissions and Doxygen Documentation

(ˢᵒᶜⁱᵉᵗʸserver) aims to be a platform for developing collaborative applications.
sTeam server project repository: sTeam.
sTeam-REST API repository: sTeam-REST

sTeam Server object permissions

sTeam command line lacks the functionality to read and set the object access permissions. The permission bits are: read,write, execute, move, insert,
annotate, sanction. The permission function was designed analogous to the getfacl() command in linux. It should display permissions as: rwxmias corresponding to the  permission granted on the object.

The the key functions are get_sanction, which returns a list of objects and permissions and sanction_object, which adds a new object and its set of permissions. The permissions is stored as an integer and the function should break the individual bits like getfact().

The permission bits for the sTeam objects are declared in the
access.h

// access.h: The permission bits

#define FAIL           -1 
#define ACCESS_DENIED   0
#define ACCESS_GRANTED  1
#define ACCESS_BLOCKED  2

#define SANCTION_READ          1
#define SANCTION_EXECUTE       2
#define SANCTION_MOVE          4
#define SANCTION_WRITE         8
#define SANCTION_INSERT       16
#define SANCTION_ANNOTATE     32

The get_sanction method defined in the access.pike returns a mapping which has the ACL(Access Control List) of all the objects in the sTeam server.


// Returns the sanction mapping of this object, if the caller is privileged
// the pointer will be returned, otherwise a copy.
final mapping
get_sanction()
{
    if ( _SECURITY->trust(CALLER) )
	return mSanction;
    return copy_value(mSanction);
}

The functions gets the permission values which are set for every object in the server.

The sanction_object method defined in the object.pike sets the permissions for the new objects.


// Set new permission for an object in the acl. Old permission are overwritten.
int sanction_object(object grp, int permission)
{
    ASSERTINFO(_SECURITY->valid_proxy(grp), "Sanction on non-proxy!");
    if ( query_sanction(grp) == permission )
      return permission; // if permissions are already fine

    try_event(EVENT_SANCTION, CALLER, grp, permission);
    set_sanction(grp, permission);

    run_event(EVENT_SANCTION, CALLER, grp, permission);
    return permission;
} 

This method makes use of the set_sanction which sets the permission onthe object. The task ahead is to make use of the above functions and write a sTeam-shell command which would provide the user to easily access and change the permissions for the objects.

Merging into the Source

The work done during GSOC 2016 by Siddhant and Ajinkya on the sTeam server was merged into the gsoc201-societyserver-devel and gsoc2016-source branches in the societyserver repository.
The merged code can be found at:

https://github.com/societyserver/sTeam/tree/gsoc2016-source
https://github.com/societyserver/sTeam/tree/gsoc2016-societyserver-devel

The merged code needs to be tested before the debian package for the sTeam server is prepared. The testing has resulted into resolving of minor bugs.

Doxygen Documentation

The documentation for the sTeam is done using doxygen. The doxygen.pike is written and used to make the documentation for the sTeam server. The Doxyfile which includes the configuration for generating the sTeam documentation is modified and input files are added. The generated documentation is deployed on the gh-pages in the societyserver/sTeam repository.
The documentation can be found at:


http://societyserver.github.io/sTeam/files.html

The header files and the constants defined are also included in the sTeam documentation.

sTeam Documentation:

SocietyserverDoc

sTeam defined constants:

SocietyServerConstants

sTeam Macro Definitions:

SocietyServerMacroDefnitions

Feel free to explore the repository. Suggestions for improvements are welcomed.

Checkout the FOSSASIA Idea’s page for more information on projects supported by FOSSASIA.

Testing, Documentation and Merging

As the GSoC period comes to an end the pressure, excitement and anxiety rises. I am working on the finishing steps of my project. I was successfully able to implement all the task I took up. A list of all the tasks and their implementation can be found here.

https://github.com/societyserver/sTeam/wiki/GSOC-2016-Work-Distribution#roadmap-and-work-distribution-on-steam-for-gsoc-2016

At the start of this week I had about 26 Pull Requests. Each Pull Request had independent pieces of code for a new task from the list. I had to merge all the pull requests and resolve the conflicts. My earlier tasks involved working on the same code so there were a lot of conflicts. I spend hours looking through the code and resolving conflicts. I also had to test each feature after merging any two of the branches. Finally we were able to combine all our code and come up with a branch that contains all the code implemented by me and Ajinkya Wavare.

https://github.com/societyserver/sTeam/tree/gsoc2016-societyserver-devel

https://github.com/societyserver/sTeam/tree/gsoc2016-source

These two are the branches we combined all our code in. I finished my work on linux command for sTeam by adding support for the last two tools which are export and import from git. I worked on to include a help to get new users to understand the use of the command.

https://github.com/societyserver/sTeam/pull/135

I also worked on documentation. I started with the testing suite which is implemented by me. I wrote comments to explain the work and also improved the code by removing unnecessary lines of code. After this I added the documentation for the new command in steam-shell that I had implemented. The command to work with groups from the steam-shell. One of the issue with the testing suite still stands unresolved. I have been breaking my head on it for a week now but to no results. I will attempt to solve it in the coming week.

https://github.com/societyserver/sTeam/issues/109

This error occurs for various objects in the first few runs and then the test suite runs normally error free.