Building a logger interface for FlightGear using Python: Part One

{ Repost from my personal blog @ https://blog.codezero.xyz/python-logger-interface-for-flightgear-part-one/ }

The FlightGear flight simulator is an open-source, multi-platform, cooperative flight simulator developed as a part of the FlightGear project. I have been using this Flight simulator for a year for Virtual Flight testing, running simulations and measuring flight parameters during various types of maneuvers. I have noticed that, logging the data, (figuring out how to log in the first place) has been quite difficult for users with less technical knowledge in such softwares.

Also, the Property Tree of FlightGear is pretty extensive making it difficult to properly traverse the huge tree to get the parameters that are actually required.

That’s when I got the idea of making a simple, easy to use, user friendly logging interface for FlightGear. I gave it a name ‘FlightGear Command Center’:wink: and the project was born at github.com/niranjan94/flightgear-cc.

After 44 commits, this is what I have now.

1. A simple dashboard to connect to FlightGear, open FlightGear with a default plane, Getting individual parameter values or to log a lot of parameters continuously

2. An interface to choose the parameters to log and the interval

  1. The User interface is a web application written in HTML/javascript.
  2. The Web application communicates with a python bridge using WebSockets.
  3. The python bridge communicates with FlightGear via telnet.
  4. The data is logged to a csv file continuously (until the user presses stop) by the bridge once the web application requests it.
The interface with FlightGear

FlightGear has an internal “telnet” command server which provides us “remote shell” into the running FlightGear process which we can exploit to interactively view or modify any property/variable of the simulation.

FlightGear can be instructed to start the server and listen for commands by passing the --telnet=socket,out,60,localhost,5555,udp command line argument while starting FlightGear. (The argument is of format --telnet=medium,direction,speed_in_hertz,localhost,PORT,style.)

Communication with that server can be done using any simple telnet interface. But FlightGear also provides us with a small wrapper class that makes retrieving and setting properties using the telnet server even more easier.

The wrapper can be obtained from the official repository atsourceforge.net/p/flightgear/flightgear/ci/master/tree/scripts/python/FlightGear.py

Using the wrapper is straightforward. Initialize an instance of the class with the hostname and port. The class will then make a connection to the telnet server.

from FlightGear import FlightGear

flightgear_server = 'localhost'  
flightgear_server_port = 5555  
fg = FlightGear(flightgear_server, flightgear_server_port)

The wrapper makes use of python’s magic methods __setitem__ and __getitem__ to make it easy for us to read or manipulate the property tree.

For example, getting the current altitude of the airplane is as easy as

print fg['/position[0]/altitude-ft']

and setting the altitude is as simple as

fg['/position[0]/altitude-ft'] = 345.2

But the important thing here is, knowing the path to the data you want in the FlightGear property tree. Most of the commonly used properties are available over at Aircraft properties reference – FlightGear Wiki.

Now that we have basic interface between python and FlightGear in place, the next step would be to setup a link between the user interface (a small web app) and the python bridge. We would be using WebSockets for that so as to have a Real-time and an always on link to the bridge which would enable us to in turn communicate with FlightGear in realtime.

We need a WebSocket server in place. So, I used the SimpleWebSocketServer.pyclass from github.com/dpallot/simple-websocket-server.

A websocket server can be created by,

from SimpleWebSocketServer import SimpleWebSocketServer, WebSocket

hostname = 'localhost'  
websocket_server_port = 8888

class SocketHandler(WebSocket):

    def handleMessage(self):
        # print the message when received 
        print self.data

    def handleConnected(self):
        print self.address, 'connected'

    def handleClose(self):
        print self.address, 'closed'

server = SimpleWebSocketServer(hostname, websocket_server_port, SocketHandler)  
server.serveforever()
  • handleMessage is called whenever a client sends a message to the server
  • handleConnected is called when a new client connects to the server
  • handleClose is called when a client disconnects from the server

A message can be sent to the clients by using the sendMessage method from within the SocketHandler.

class SocketHandler(WebSocket):

    def handleMessage(self):
        # send a hello whenever a message is received  
        print self.data
        self.sendMessage('Hello')

    def handleConnected(self):
        print self.address, 'connected'

    def handleClose(self):
        print self.address, 'closed'

We now have a WebSocket server in place. Now the web app can easily talk to this server using javascript websockets API. Which would be continued in upcoming blog articles.

Building interactive elements with HTML and javascript: Interact.js + resizing

{ Repost from my personal blog @ https://blog.codezero.xyz/building-interactive-elements-with-html-and-javascript-interact-js-resizing/ }

In a few of the past blog posts, we saw about implementing resizing with HTML and javascript. The functionality was pretty basic with simple resizing. In the last blog post we saw about interact.js.

interact.js is a lightweight, standalone JavaScript module for handling single-pointer and multi-touch drags and gestures with powerful features including inertia and snapping.

Getting started with Interact.js

You have multiple option to include the library in your project.

  • You can use bower to install (bower install interact) (or)
  • npm (npm install interact.js) (or)
  • You could directly include the library from a CDN (https://cdnjs.cloudflare.com/ajax/libs/interact.js/1.2.6/interact.min.js).
Implementing resizing

Let’s create a simple box using HTML. We’ll add a class called resizable to it so that we can reference it to initialize Interact.js

<div class="resizable">  
    Use right/bottom edge to resize
</div>

We need to create an interact instance. Once the instance is created, we have to call the resizable method on it to add resize support to the div.

interact('.resizable')
  .resizable({
    edges: { right: true, bottom: true }
  })
  .on('resizemove', function (event) {
    

  });

Inside the resizable method, we can pass configuration options. The edgesconfig key allows us to specify on which all edges, resizing should be allowed. Right now, we have allowed on the right and bottom edges. Similarly we can have resizing support in the top and left edges too.

The resizemove event is triggered by interact every time the user tries to resize the div. From the event, we can get the box that is being resized, (i.e) the target by accessing event.target.

The event object also provides us event.rect.width and event.rect.height which is the width and height of the div after resizing. We’ll not set this as the width of the div so that, the user is able to see the width change.

var target = event.target;
    // update the element's style
    target.style.width  = event.rect.width + 'px';
    target.style.height = event.rect.height + 'px';

We can also instruct Interact.js to preserve the aspect ratio of the box by adding an option preserveAspectRatio: true to the configuration object passed to resizable method during initialization.

JavaScript
interact('.resizable')
  .resizable({
    edges: { right: true, bottom: true }
  })
  .on('resizemove', function (event) {
    var target = event.target;

    // update the element's style
    target.style.width  = event.rect.width + 'px';
    target.style.height = event.rect.height + 'px';
  });

Resizing and drag-drop (with Interact.js) were used to create the Scheduler tool at Open Event. The tool allows event/track organizers to easily arrange the sessions into their respective rooms by drag-drop and also to easily change the timings of the events by resizing the event block. The entire source code of the scheduler can be viewed at app/static/js/admin/event/scheduler.js in the Open Event Organizer server’s GitHub repository.

Demo:
https://jsfiddle.net/xdfocdty/

Getting fired up with Firebase Database

As you might’ve noticed, in my Open Event Android Project, we are asking the user to enter his/her details and then using these details at the backend for generating the app according to his/her needs.

One thing to wonder is how did we transmit the details from webpage to the server.

Well, this is where Firebase comes to the rescue one more time!

If you’ve read my previous post on Firebase Storage, you might have started to appreciate what an awesometastic service Firebase is.

So without any further adieu, lets get started with this.

Step 1 :

Add your project to Firebase from the console.

 newProj
Click on the Blue button

Step 2 :

Add Firebase to your webapp

Open the project, you’ve just created and click on the bright red button that says, “ Add Firebase to your web app”

 addFirebase

Copy the contents from here and paste it after your HTML code.

Step 3 :

Next up, navigate to the Database section in your console and move to the Rules tab.

 screenshot-area-2016-07-18-204133.png

For now, let us edit the rules to allow anyone to read and write to the database.

 screenshot-area-2016-07-18-204437

Almost all set up now.

Step 4 :

Modify the HTML to allow entering data by the user

This looks something like this :

Now let us setup our javascript to extract this data and store this in Firebase Database.

We are almost finished with uploading the data to the database.

Enter data inside the fields and press submit.

If everything went well, you will be able to see the newly entered data inside your database.

screenshot-area-2016-07-18-205651.png

Now on to retrieving this data on the server.

Our backend runs on a python script, so we have a library known as python-firebase which helps us easily fetch the data stored in the Firebase database.

The code for it goes like this

The data will be returned in JSON format, so you can manipulate and store it as you wish.

Well, that’s it!

You now know how to store and retrieve data to and from Firebase.
It makes the work a lot simpler as there is no Database schema or tables that need to be defined, firebase handles this on its own.

I hope that you found this tutorial helpful, and if you have any doubts regarding this feel free to comment down below, I would love to help you out.

Cheers.

Push your apk to your GitHub repository from Travis

In this post I’ll guide you on how to directly upload the compiled .apk from travis to your GitHub repository.

Why do we need this?

Well, assume that you need to provide an app to your testers after each commit on the repository, so instead of manually copying and emailing them the app, we can setup travis to upload the file to our repository where the testers can fetch it from.

So, lets get to it!

Step 1 :

Link Travis to your GitHub Account.

Open up https://travis-ci.org.

Click on the green button in the top right corner that says “Sign in with GitHub”

screenshot-area-2016-07-15-205733.png<

Step 2 :

Add your existing repository to Travis

Click the “+” button next to your Travis Dashboard located on the left.

screenshot-area-2016-07-15-210630.png<

Choose the project that you want to setup Travis from the next page

screenshot-area-2016-07-15-210916.png
Toggle the switch for the project that you want to integrate

Click the cog here and add an Environment Variable named GITHUB_API_KEY.
Proceed by adding your Personal Authentication Token there.
Read up here on how to get the Token.

 screenshot-area-2016-07-15-213931.png<

Great, we are pretty much done here.

Let us move to the project repository that we just integrated and create a new file in the root of repository by clicking on the “Create new file” on the repo’s page.

Name it .travis.yml and add the following commands over there

language: android 
jdk:
  - oraclejdk8
android:
  components:
    - tools
    - build-tools-24.0.0
    - android-24
    - extra-android-support
    - extra-google-google_play_services
    - extra-android-m2repository
    - extra-google-m2repository
    - addon-google_apis-google-24
 before_install:
 - chmod +x gradlew
 - export JAVA8_HOME=/usr/lib/jvm/java-8-oracle
 - export JAVA_HOME=$JAVA8_HOME
 after_success:
 - chmod +x ./upload-gh-pages.sh
 - ./upload-apk.sh
 script:
 - ./gradlew build

Next, create a bash file in the root of your repository using the same method and name it upload-apk.sh

  #create a new directory that will contain out generated apk
  mkdir $HOME/buildApk/ 
  #copy generated apk from build folder to the folder just created
  cp -R app/build/outputs/apk/app-debug.apk $HOME/android/
  #go to home and setup git
  cd $HOME
  git config --global user.email "[email protected]"
  git config --global user.name "Your Name" 
  #clone the repository in the buildApk folder
  git clone --quiet --branch=master  https://user-name:[email protected]/user-name/repo-name  master > /dev/null
  #go into directory and copy data we're interested
  cd master  cp -Rf $HOME/android/* .
  #add, commit and push files
  git add -f .
  git remote rm origin
  git remote add origin https://user-name:[email protected]/user-name/repo-name.git
  git add -f .
  git commit -m "Travis build $TRAVIS_BUILD_NUMBER pushed"
  git push -fq origin master > /dev/null
  echo -e "Donen"

Once you have done this, commit and push these files, a Travis build will be initiated in few seconds.
You can see it ongoing in your Dashboard at https://travis-ci.org/.

After the build has completed, you will can see an app-debug.apk in your Repository.

IMPORTANT NOTE :

You might be wondering as to why did I write [skip ci] in the commit message.

Well the reason for that is, Travis starts a new build as soon as it detects a commit made on the master branch of your repository.

So once the apk is uploaded, that will trigger another build in Travis and hence forming an infinite loop.

We can prevent this in 2 ways :

First, simply write [skip ci] somewhere in the commit message and it will cause Travis to ignore the commit.

Or, push the apk to any other branch which is not configured for Travis build.

So well, that’s almost it.

I hope that you found this tutorial helpful, and if you have any doubts regarding this feel free to comment down below, I would love to help you out.

Cheers.

Ideas on using Celery with Flask for background tasks

Simply put, Celery is a background task runner. It can run time-intensive tasks in the background so that your application can focus on the stuff that matters the most. In context of a Flask application, the stuff that matters the most is listening to HTTP requests and returning response.

By default, Flask runs on a single-thread. Now if a request is executed that takes several seconds to run, then it will block all other incoming requests as it is single-threaded. This will be a very bad-experience for the user who is using the product. So here we can use Celery to move time-hogging part of that request to the background.

I would like to let you know that by “background”, Celery means another process. Celery starts worker processes for the running application and these workers receive work from the main application. Celery requires a broker to be used. Broker is nothing but a database that stores results of a celery task and provides a shared interface between main process and worker processes. The output of the work done by the workers is stored in the Broker. The main application can then access these results from the Broker.

Using Celery to set background tasks in your application is as simple as follows –

@celery.task
def background_task(*args, **kwargs):
    # do stuff
    # more stuff

Now the function background_task becomes function-able as a background task. To execute it as a background task, run –

task = background_task.delay(*args, **kwargs)
print task.state  # task current state (PENDING, SUCCESS, FAILURE)

Till now this may look nice and easy but it can cause lots of problems. This is because the background tasks run in different processes than the main application. So the state of the worker application differs from the real application.

One common problem because of this is the lack of request context. Since a celery task runs in a different process, so the request context is not available. Therefore the request headers, cookies and everything else is not available when the task actually runs. I too faced this problem and solved it using an excellent snippet I found on the Internet.

"""
Celery task wrapper to set request context vars and global
vars when a task is executed
Based on http://xion.io/post/code/celery-include-flask-request-context.html
"""
from celery import Task
from flask import has_request_context, make_response, request, g

from app import app  # the flask app


__all__ = ['RequestContextTask']


class RequestContextTask(Task):
    """Base class for tasks that originate from Flask request handlers
    and carry over most of the request context data.
    This has an advantage of being able to access all the usual information
    that the HTTP request has and use them within the task. Pontential
    use cases include e.g. formatting URLs for external use in emails sent
    by tasks.
    """
    abstract = True

    #: Name of the additional parameter passed to tasks
    #: that contains information about the original Flask request context.
    CONTEXT_ARG_NAME = '_flask_request_context'
    GLOBALS_ARG_NAME = '_flask_global_proxy'
    GLOBAL_KEYS = ['user']

    def __call__(self, *args, **kwargs):
        """Execute task code with given arguments."""
        call = lambda: super(RequestContextTask, self).__call__(*args, **kwargs)

        # set context
        context = kwargs.pop(self.CONTEXT_ARG_NAME, None)
        gl = kwargs.pop(self.GLOBALS_ARG_NAME, {})

        if context is None or has_request_context():
            return call()

        with app.test_request_context(**context):
            # set globals
            for i in gl:
                setattr(g, i, gl[i])
            # call
            result = call()
            # process a fake "Response" so that
            # [email protected]_request`` hooks are executed
            # app.process_response(make_response(result or ''))

        return result

    def apply_async(self, args=None, kwargs=None, **rest):
        self._include_request_context(kwargs)
        self._include_global(kwargs)
        return super(RequestContextTask, self).apply_async(args, kwargs, **rest)

    def apply(self, args=None, kwargs=None, **rest):
        self._include_request_context(kwargs)
        self._include_global(kwargs)
        return super(RequestContextTask, self).apply(args, kwargs, **rest)

    def retry(self, args=None, kwargs=None, **rest):
        self._include_request_context(kwargs)
        self._include_global(kwargs)
        return super(RequestContextTask, self).retry(args, kwargs, **rest)

    def _include_request_context(self, kwargs):
        """Includes all the information about current Flask request context
        as an additional argument to the task.
        """
        if not has_request_context():
            return

        # keys correspond to arguments of :meth:`Flask.test_request_context`
        context = {
            'path': request.path,
            'base_url': request.url_root,
            'method': request.method,
            'headers': dict(request.headers),
        }
        if '?' in request.url:
            context['query_string'] = request.url[(request.url.find('?') + 1):]

        kwargs[self.CONTEXT_ARG_NAME] = context

    def _include_global(self, kwargs):
        d = {}
        for z in self.GLOBAL_KEYS:
            if hasattr(g, z):
                d[z] = getattr(g, z)
        kwargs[self.GLOBALS_ARG_NAME] = d

To run a task in Request context mode, do –

@celery.task(base=RequestContextTask, bind=True)
def background_task(self, *args, **kwargs):
    # do stuff
    # more stuff

If you are wondering what the RequestContextTask class does, it simply stores all request context vars and global vars when a background task is called (task.delay()) and then unpacks those values to their proper places when the task is about to be run. The above snippet can be easily extended to store any value.

Another challenge that some people may face is the occasional Parsing/Serialization error. This happens because the data being sent to/from a function that is to be background executed is too complex.

Serialization is the process of converting complex data structures and objects into a plain string. Serialization of data is necessary because the background tasks and the main thread run in different processes. Now think how will the main thread communicate the celery thread to do some task. This is done using serialization of the concerned data. So to avoid serialization errors, it is recommended that you make background tasks such that they require only simple arguments to run and they return only simple data.

So basically keeping small and simple tasks is recommended when using Celery. Follow this golden rule and you will not run into any problems.

 

{{ Repost from my personal blog | http://aviaryan.in/blog/gsoc/celery-flask-good-ideas.html }}

Building interactive elements with HTML and javascript: Interact.js + drag-drop

{ Repost from my personal blog @ https://blog.codezero.xyz/building-interactive-elements-with-html-and-javascript-interact-js-drag-drop }

In a few of the past blog posts, we saw about implementing drag-drop andresizing with HTML and javascript. The functionality was pretty basic with a simple drag-and-drop and resizing. That is where, a javascript library called as interact.js comes in.

interact.js is a lightweight, standalone JavaScript module for handling single-pointer and multi-touch drags and gestures with powerful features including inertia and snapping.

With Interact.js, building interactive elements is like a eating a piece of your favorite cake – that easy !

Getting started with Interact.js

You have multiple option to include the library in your project.

  • You can use bower to install (bower install interact) (or)
  • npm (npm install interact.js) (or)
  • You could directly include the library from a CDN (https://cdnjs.cloudflare.com/ajax/libs/interact.js/1.2.6/interact.min.js).

Implementing a simple draggable

Let’s start with some basic markup. We’ll be using the draggable class to enable interact.js on this element.

<div id="box-one" class="draggable">  
  <p> I am the first Box </p>
</div>  
<div id="box-two" class="draggable">  
    <p> I am the second Box </p>
</div>

The first step in using interact.js is to create an interact instance. Which you can create by using interact('<the selector>'). Once the instance is created, you’ll have to call the draggable method on it to enable drag. Draggable accepts a javascript object with some configuration options and some pretty useful callbacks.

// target elements with the "draggable" class
interact('.draggable')  
  .draggable({
    // enable inertial throwing
    inertia: true,
    // keep the element within the area of it's parent
    restrict: {
      restriction: "parent",
      endOnly: true,
      elementRect: { top: 0, left: 0, bottom: 1, right: 1 }
    },
    // enable autoScroll
    autoScroll: true,
    // call this function on every dragmove event
    onmove: dragMoveListener,
  });

  function dragMoveListener (event) {
    var target = event.target,
        // keep the dragged position in the data-x/data-y attributes
        x = (parseFloat(target.getAttribute('data-x')) || 0) + event.dx,
        y = (parseFloat(target.getAttribute('data-y')) || 0) + event.dy;

    // translate the element
    target.style.webkitTransform =
    target.style.transform =
      'translate(' + x + 'px, ' + y + 'px)';

    // update the posiion attributes
    target.setAttribute('data-x', x);
    target.setAttribute('data-y', y);
  }

Here we use the onmove event to move the box according to the dx and dyprovided by interact when the element is dragged.

Implementing a simple drag-drop

Now to the above draggable, we’ll add a drop zone into which the two draggable boxes can be dropped.

<div id="dropzone" class="dropzone">You can drop the boxes here</div>

Similar to a draggable, we first create an interact instance. Then we call the dropzone method on to tell interact that, that div is to be considered as a dropzone. The dropzone method accepts a json object with configuration options and callbacks.

// enable draggables to be dropped into this
interact('.dropzone').dropzone({  
  // Require a 50% element overlap for a drop to be possible
  overlap: 0.50,

  // listen for drop related events:

  ondropactivate: function (event) {
    // add active dropzone feedback
    event.target.classList.add('drop-active');
  },
  ondragenter: function (event) {
    var draggableElement = event.relatedTarget,
        dropzoneElement = event.target;

    // feedback the possibility of a drop
    dropzoneElement.classList.add('drop-target');
  },
  ondragleave: function (event) {
    // remove the drop feedback style
    event.target.classList.remove('drop-target');
  },
  ondrop: function (event) {
    event.relatedTarget.textContent = 'Dropped';
  },
  ondropdeactivate: function (event) {
    // remove active dropzone feedback
    event.target.classList.remove('drop-active');
    event.target.classList.remove('drop-target');
  }
});

Each event provides two important properties. relatedTarget which gives us the DOM object of the draggable that is being dragged. And target which gives us the DOM object of the dropzone. We can use this to provide visual feedback for the user when he/she is dragging.

Demo:

https://jsfiddle.net/niranjan94/yqwc4hqz/2/

Set up Firebase to upload user files

If you’ve read my previous post on uploading files to server, you might have noticed that it was not an easy task to achieve.

There is way too much boilerplate code for uploading a single file, and it will be much more complex in case we plan to upload multiple files, which I need to do for my project.

So what’s the solution to this?

ENTER FIREBASE!

Yeah, you read it right, Firebase once again to the rescue!

I came to know that firebase allows user to store and access the files easily to the built in storage.

Enough chatter for now, lets get to code!

Step 1 :

Add your project to Firebase from the console.

newProj.PNG

Click on the Blue button

Step 2 :

Add Firebase to your webapp

Open the project, you’ve just created and click on the bright red button that says, “ Add Firebase to your web app”

addFirebase.PNGCopy the contents from here and paste it after your HTML code.

Step 3 :

Open the “Storage” tab from the navigation drawer and navigate to the rules tab over there.
We need to set up specific rules as to who all can upload and read files to the storage bucket.

storageRules.PNG
For testing purposes, I’ve allowed everyone to read and write to my storage, but that shouldn’t be the case for your production app

Step 4 :

Add code for uploading your files.

First create a document selection widget and an upload button in your website’s index.html.

<tr>
<td valign=”top”>
<label for=”icon”>Zip File</label>
</td>
<td valign=”top”>
<input accept=”.zip” type=”file” id=”uploadZip” name=”icon”>
</td>
</tr>

Next, create a button to initiate the upload

<tr>
<td colspan=”5″ style=”text-align:center”>
<button type=”submit”>Upload Zip</button>
</td>
</tr>

Next up, inside the JavaScript, add a submitlistener for the submit button and call preventDefault inside it to prevent the form from doing the default action.

var form = document.querySelector(“form”);
form.addEventListener(“submit”, function(event) {
event.preventDefault();

Next up, get a reference to the upload location from your firebase storage bucket.

var timestamp = Number(new Date());
var storageRef = firebase.storage().ref(timestamp.toString());

Next, get the upload button from its ID and add its contents to a variable named file_data.

var $ = jQuery;
var file_data = $(‘#uploadZip’).prop(‘files’)[0];

Now upload that file to firebase.

storageRef.put(file_data);

If everything went as expected, you’ll be able to see the uploaded files onto your firebase console.

 storage

So, you can really appreciate the awesomeness of Firebase by now.
It has replaced the work done by over 50+ lines of code (spread around AJAX calls, PHP Scripts and JavaScript methods) by a single method call.

I would urge you to go through the documentation for more clarity on this.
https://firebase.google.com/docs/storage/

Well, that was it for now.
Next time, I’ll tell you how to retrieve the files back from the storage and add user’s details to Firebase Database.(Yeah, no need for Tables and SQL anymore!)

Cheers. 😀

Uploading a file to a server via PHP

Uploading a file to a server via PHP

If you have been following my posts about my GSoC project, you would be knowing that we are making an app generator which will allow users to easily generate an android app for any event that they plan to host.

So, the next thing that we wanted in our app was to allow the users to upload a zip containing the json files (in case they don’t have an API, from where app can fetch data from) and then upload it to the server where we can use these files during the app compilation.

Steps below will tell you , how we achieved it :

Changes to HTML

First thing that we needed to do was add a file upload element to out HTML page that would allow only .zip files to be uploaded.
It was pretty simple one liner code which goes as follows

<tr>
<td valign=”top”>
<label for=”sessions”>Zip containing .json files</label>
</td>
<td valign=”top”>
<input accept=”.zip” type=”file” id=”uploadZip” name=”sessions”>
</td>
</tr>

PHP script to upload file on to the server

Next, we needed a server sided script (I used PHP) which would upload the zip provided by the user on to the server and store it to a unique location for each user.
The code for that was,

<?php
if ( 0 < $_FILES[‘file’][‘error’] ) {
echo ‘Error: ‘ . $_FILES[‘file’][‘error’] . ‘<br>’;
}
else {
move_uploaded_file($_FILES[‘file’][‘tmp_name’],“/var/www/html/uploads/upload.zip”);
}
?>

So what is happening here is basically the input arg. is first checked whether it is null or not null.
If it is null, and error is thrown back to the user, else the file is renamed and uploaded to the uploads folder in the server’s public directory.

Changes to the JavaScript

This was the part that needed most of the changes to be done, we first had to store the file that is to be uploaded in the form data, and then make and AJAX call to the php file located on the server.

var file_data = $(‘#uploadZip’).prop(‘files’)[0];
var form_data = new FormData();
form_data.append(‘file’, file_data);
$.ajax(
{ url: ‘/upload.php’, // point to server-side PHP script
cache: false,
contentType: false,
processData: false,
data: form_data,
type: ‘post’,
success: function(php_script_response){
ajaxCall1(); } //Chain up another AJAX call for further operations
});

So, that’s almost it!
Some server sided changes were also required like allowing the web user to execute the upload.php script and making the uploads directory writable by the web user.

Well, does it work?

Um, yeah it does.
There are a few issues with concurrent users which we are still debugging, but apart from that it works like a charm!

Here you can see a folder created by each user based on his/her timestamp
And here you can see the file that was uploaded y him/her
screenshot-area-2016-07-04-124957
Lastly our webapp (Looks stunning right?)

So, that was all for this week, hope to see you again next time.
Cheers and all the best 🙂

Implementing revisioning feature in Open Event

{ Repost from my personal blog @ https://blog.codezero.xyz/implementing-revisioning-feature-in-open-event }

As I said in my previous blog post about Adding revisioning to SQLAlchemy Models,

In an application like Open Event, where a single piece of information can be edited by multiple users, it’s always good to know who changed what. One should also be able to revert to a previous version if needed.

Let’s have a quick run through on how we can enable SQLAlchemy-Continuum on our project.

  1. Install the library SQLAlchemy-Continuum with pip
  2. Add __versioned__ = {} to all the models that need to be versioned.
  3. Call make_versioned() before the models are defined
  4. Call configure_mappers from SQLAlchemy after declaring all the models.

Example:

import sqlalchemy as sa  
from sqlalchemy_continuum import make_versioned

# Must be called before defining all the models
make_versioned()

class Event(Base):

    __tablename__ = 'events'
    __versioned__ = {}  # Must be added to all models that are to be versioned

    id = sa.Column(sa.Integer, primary_key=True, autoincrement=True)
    name = sa.Column(sa.String)
    start_time = sa.Column(db.DateTime, nullable=False)
    end_time = sa.Column(db.DateTime, nullable=False)
    description = db.Column(db.Text)
    schedule_published_on = db.Column(db.DateTime)

# Must be called after defining all the models
sa.orm.configure_mappers()

We have SQLAlchemy-Continuum enabled now. You can do all the read/write operations as usual. (No change there).

Now, for the part where we give the users an option to view/restore revisions. The inspiration for this, comes from wordpress’s wonderful revisioning functionality.

The layout is well designed. The differences are shown in an easy-to-read form. The slider on top makes it intuitive to move b/w revisions. We have a Restore this Revision button on the top-right to switch to that revision.

A similar layout is what we would like to achieve in Open Event.

  1. A slider to switch b/w sessions
  2. A pop-over infobox on the slider to show who made that change
  3. A button to switch to that selected revision.
  4. The colored-differences shown in side-by-side manner.

To make all this a bit easier, SQLAlchemy-Continuum provides us with some nifty methods.

count_versions is a method that allows us to know the number of revisions a record has.

event = session.query(Event).get(1)  
count = count_versions(event)  # number of versions of that event

Next one is pretty cool. All the version objects have a property called as changeset which holds a dict of changed fields in that version.

event = Event(name=u'FOSSASIA 2016', description=u'FOSS Conference in Asia')  
session.add(article)  
session.commit(article)

version = event.versions[0]  # first version  
version.changeset  
# {
#   'id': [None, 1],
#   'name': [None, u'FOSSASIA 2016'],
#   'description': [None, u'FOSS Conference in Asia']
# }

event.name = u'FOSSASIA 2017'  
session.commit()

version = article.versions[1]  # second version  
version.changeset  
# {
#   'name': [u'FOSSASIA 2016', u'FOSSASIA 2017'],
# }

As you can see, dict holds the fields that changed and the content the changed (before and after). And this is what we’ll be using for generating those pretty diffs that the guys and girls over at wordpress.com have done. And for this we’ll be using two things.

  1. A library named diff-match-patch. It is a library from Google which offers robust algorithms to perform the operations required for synchronizing plain text.
  2. A small recipe from from code.activestate.com Line-based diffs with the necessary HTML markup for styling insertions and deletions.
import itertools  
import re

import diff_match_patch

def side_by_side_diff(old_text, new_text):  
    """
    Calculates a side-by-side line-based difference view.

    Wraps insertions in <ins></ins> and deletions in <del></del>.
    """
    def yield_open_entry(open_entry):
        """ Yield all open changes. """
        ls, rs = open_entry
        # Get unchanged parts onto the right line
        if ls[0] == rs[0]:
            yield (False, ls[0], rs[0])
            for l, r in itertools.izip_longest(ls[1:], rs[1:]):
                yield (True, l, r)
        elif ls[-1] == rs[-1]:
            for l, r in itertools.izip_longest(ls[:-1], rs[:-1]):
                yield (l != r, l, r)
            yield (False, ls[-1], rs[-1])
        else:
            for l, r in itertools.izip_longest(ls, rs):
                yield (True, l, r)

    line_split = re.compile(r'(?:r?n)')
    dmp = diff_match_patch.diff_match_patch()

    diff = dmp.diff_main(old_text, new_text)
    dmp.diff_cleanupSemantic(diff)

    open_entry = ([None], [None])
    for change_type, entry in diff:
        assert change_type in [-1, 0, 1]

        entry = (entry.replace('&', '&amp;')
                      .replace('<', '&lt;')
                      .replace('>', '&gt;'))

        lines = line_split.split(entry)

        # Merge with previous entry if still open
        ls, rs = open_entry

        line = lines[0]
        if line:
            if change_type == 0:
                ls[-1] = ls[-1] or ''
                rs[-1] = rs[-1] or ''
                ls[-1] = ls[-1] + line
                rs[-1] = rs[-1] + line
            elif change_type == 1:
                rs[-1] = rs[-1] or ''
                rs[-1] += '<ins>%s</ins>' % line if line else ''
            elif change_type == -1:
                ls[-1] = ls[-1] or ''
                ls[-1] += '<del>%s</del>' % line if line else ''

        lines = lines[1:]

        if lines:
            if change_type == 0:
                # Push out open entry
                for entry in yield_open_entry(open_entry):
                    yield entry

                # Directly push out lines until last
                for line in lines[:-1]:
                    yield (False, line, line)

                # Keep last line open
                open_entry = ([lines[-1]], [lines[-1]])
            elif change_type == 1:
                ls, rs = open_entry

                for line in lines:
                    rs.append('<ins>%s</ins>' % line if line else '')

                open_entry = (ls, rs)
            elif change_type == -1:
                ls, rs = open_entry

                for line in lines:
                    ls.append('<del>%s</del>' % line if line else '')

                open_entry = (ls, rs)

    # Push out open entry
    for entry in yield_open_entry(open_entry):
        yield entry

So, what we have to do is,

  1. Get the changeset from a version
  2. Run each field’s array containing the old and new text through the side_by_side_diff method.
  3. Display the output on screen.
  4. Use the markups <ins/> and <del/> to style changes.

So, we do the same for each version by looping through the versions array accessible from an event record.

For the slider, noUiSlider javascript library was used. Implementation is simple.

<div id="slider"></div>

<script type="text/javascript">  
    $(function () {
        var slider = document.getElementById('slider');
        noUiSlider.create(slider, {
            start: [0],
            step: 1,
            range: {
                'min': 0,
                'max': 5
            }
        });
    });
</script>

This would create a slider that can go from 0 to 5 and will start at position 0.

By listening to the update event of the slider, we’re able to change which revision is displayed.

slider.noUiSlider.on('update', function (values, handle) {  
    var value = Math.round(values[handle]);
    // the current position of the slider
    // do what you have to do to change the displayed revision
});

And to get the user who caused a revision, you have to access the user_idparameter of the transaction record of a particular version.

event = session.query(Event).get(1)  
version_one = event.versions[0]  
transaction = transaction_class(version_one)  
user_id = transaction.user_id

So, with the user ID, you can query the user database to get the user who made that revision.

The user_id is automatically populated if you’re using Flask, Flask-login and SQLAlchemy-Continuum’s Flask Plugin. Enabling the plugin is easy.

from sqlalchemy_continuum.plugins import FlaskPlugin  
from sqlalchemy_continuum import make_versioned

make_versioned(plugins=[FlaskPlugin()])

This is not a very detailed blog post. If you would like to see the actual implementation, you can checkout the Open Event repository over at GitHub. Specifically, the file browse_revisions.html.

The result is,

Still needs some refinements in the UI. But, it gets the job done :wink:

Introduction to JWT

In this post, I will try to explain what is JWT, what are its advantages and why you should be using it.

JWT stands for JSON Web Tokens. Let me explain what each word means.

  1. Tokens – Token is in tech terms a piece of data (claim) which gives access to certain piece of information and allows certain actions.
  2. Web – Web here means that it was designed to be used on the web i.e. web projects.
  3. JSON – JSON means that the token can contain json data. In JWT, the json is first serialized and then Base64 encoded.

A JWT looks like a random sequence of strings separated by 2 dots. The yyyyy part which you see below has the Base64 encoded form of json data mentioned earlier.

xxxxx.yyyyy.zzzzz

The 3 parts in order are –

  • Header – Header is the base64 encoded json which contains hashing algorithm on which the token is secured.
  • Payload – Payload is the base64 encoded json data which needs to be shared through the token. The json can include some default keys like iss (issuer), exp (expiration time), sub (subject) etc. Particularly exp here is the interesting one as it allows specifying expiry time of the token.

At this point you might be thinking that how is JWT secure if all we are doing is base64 encoding payload. After all, there are easy ways to decode base64. This is where the 3rd part (zzzzz) is used.

  • Signature – Signature is a hashed string made up by the first two parts of the token (header and payload) and a secret. The secret should be kept confidential to the owner who is authenticating using JWT. This is how the signature is created. (assuming HMACSHA256 as the algorithm)
HMACSHA256(
  xxxxx + "." + yyyyy,
  secret)

How to use JWT for authentication

Once you realize it, the idea of JWT is quite simple. To use JWT for authentication, what you do is you make the client POST their username and password to a certain url. If the combination is correct, you return a JWT including username in the “Payload”. So the payload looks like –

{
  "username": "john.doe"
}

Once the client has this JWT, they can send the same in Header when accessing protected routes. The server can read the JWT from the header and verify its correctness by matching the signature (zzzzz part) with the encoded hash created using header+payload and secret (generated signature). If the strings match, it means that the JWT is valid and therefore the request can be given access to the routes. BTW, you won’t have to go through such a deal for using JWT for authentication, there are already a handful of libraries that can do these for you.

Why use JWT over auth tokens ?

As you might have noticed in the previous section, JWT has a payload field that can contain any type of information. If you include username in it, you will be able to identify the user just by validating the JWT and there will be no need to read from the database unlike typical tokens which require a database read cycle to get the claimed user. Now if you go ahead and include permission informations in JWT too (like'isAdmin': True), then more database reads can be prevented. And this optimization comes at no cost at all. So this is why you should be using JWT.

We at Open Event use JWT for our primary means of authentication. Apart from that, we support basic authentication too. Read this post for some points about that.

That’s it for now. Thanks for reading.

 

{{ Repost from my personal blog http://aviaryan.in/blog/gsoc/jwt-intro.html }}