Open Event

Getting fired up with Firebase Database

As you might’ve noticed, in my Open Event Android Project, we are asking the user to enter his/her details and then using these details at the backend for generating the app according to his/her needs.

One thing to wonder is how did we transmit the details from webpage to the server.

Well, this is where Firebase comes to the rescue one more time!

If you’ve read my previous post on Firebase Storage, you might have started to appreciate what an awesometastic service Firebase is.

So without any further adieu, lets get started with this.

Step 1 :

Add your project to Firebase from the console.

 newProj
Click on the Blue button

Step 2 :

Add Firebase to your webapp

Open the project, you’ve just created and click on the bright red button that says, “ Add Firebase to your web app”

 addFirebase

Copy the contents from here and paste it after your HTML code.

Step 3 :

Next up, navigate to the Database section in your console and move to the Rules tab.

 screenshot-area-2016-07-18-204133.png

For now, let us edit the rules to allow anyone to read and write to the database.

 screenshot-area-2016-07-18-204437

Almost all set up now.

Step 4 :

Modify the HTML to allow entering data by the user

This looks something like this :

<form name="htmlform" id="form" enctype="multipart/form-data">
<p align="center"><b><big>FOSSASIA's App Generator</big></b></p>
<table align="center"
width = "900px"
height="200px">
<tr>
<td valign="top">
<label for="Email">Email</label>
</td>
<td valign="top">
<input id="email" type="email" name="Email" size="30">
</td>
<td>
<td valign="top">
<label for="name">App's Name</label>
</td>&nbsp;
<td valign="top">
<input id="appName" type="text" name="App_Name" maxlength="50" size="30">
</td>&nbsp;
</tr>
<tr>
<td valign="top">
<label for="link">Api Link</label>
</td>
<td valign="top">
<input id="apiLink" type="url" name="Api_Link" maxlength="90" size="30">
</td>
</tr>
<tr>
<td valign="top">
<label for="sessions">Zip containing .json files</label>
</td>
<td valign="top">
<input accept=".zip" type="file" id="uploadZip" name="sessions">
</td>
</tr>
<tr>
<td colspan="5" style="text-align:center">
<button type="submit">Generate and Download app</button>
</td>
</tr>
</table>
</form>
view raw index.html hosted with ❤ by GitHub

Now let us setup our javascript to extract this data and store this in Firebase Database.

<script src="https://www.gstatic.com/firebasejs/live/3.0/firebase.js"></script>
<script src="https://code.jquery.com/jquery-1.10.2.js"></script>
<script src="https://code.jquery.com/ui/1.11.2/jquery-ui.js"></script>
<script>
var $ = jQuery;
var timestamp = Number(new Date()); //this will server as a unique ID for each user
var form = document.querySelector("form");
var config = {
apiKey: "API_KEY",
authDomain: "app-id.firebaseapp.com",
databaseURL: "https://app-id.firebaseio.com",
storageBucket: "app-id.appspot.com",
};
firebase.initializeApp(config);
var database = firebase.database();
form.addEventListener("submit", function(event) {
event.preventDefault();
var ary = $(form).serializeArray();
var obj = {};
for (var a = 0; a < ary.length; a++) obj[ary[a].name] = ary[a].value;
console.log("JSON",obj);
var file_data = $('#uploadZip').prop('files')[0];
var storageRef = firebase.storage().ref(timestamp.toString());
storageRef.put(file_data);
var form_data = new FormData();
form_data.append('file', file_data);
firebase.database().ref('users/' + timestamp).set(obj);
database.ref('users/' + timestamp).once('value').then(function(snapshot) {
console.log("Received value",snapshot.val());
)};
});
</script>
view raw script.js hosted with ❤ by GitHub

We are almost finished with uploading the data to the database.

Enter data inside the fields and press submit.

If everything went well, you will be able to see the newly entered data inside your database.

screenshot-area-2016-07-18-205651.png

Now on to retrieving this data on the server.

Our backend runs on a python script, so we have a library known as python-firebase which helps us easily fetch the data stored in the Firebase database.

The code for it goes like this

firebase = firebase.FirebaseApplication('https://app-id.firebaseio.com', None)
result = firebase.get('/users', str(arg))
jsonData = json.dumps(result)
email = json.dumps(result['Email'])
email = email.replace('"', '')
app_name = json.dumps(result['App_Name'])
app_name = app_name.replace('"', '')
print app_name
print email
view raw firebase.py hosted with ❤ by GitHub

The data will be returned in JSON format, so you can manipulate and store it as you wish.

Well, that’s it!

You now know how to store and retrieve data to and from Firebase.
It makes the work a lot simpler as there is no Database schema or tables that need to be defined, firebase handles this on its own.

I hope that you found this tutorial helpful, and if you have any doubts regarding this feel free to comment down below, I would love to help you out.

Cheers.

Continue ReadingGetting fired up with Firebase Database

Adding Client Side validation to Login and Registration Forms

Its very important to have a client side validation apart from a server side validation. The server side validation only helps the developers but not the clients. Thus it was necessary to add a client side validation to the Login page and its other components so that the users  would be comfortable in this.

I had never before done this and was looking at how to achieve this using jQuery or JavaScript when I cam to know that we were already using an amazing validation tool : Bootstrap Validator

It just involves wrapping the form in the html with the validator plugin and all the checks are automatically carried out by it.

 

1.png

As you can see in the above image, we have just added a data-toggle=”validator” line to the form which automatically wraps the form with the plugin. The above form is the Create New Password form which checks whether the new password and the password entered again are matching or not. If not then the line ,

data-match="#new_password"

checks it with the password in the new_password field and gives an error on the page to the client defined by the following,

data-error="Passwords do not match

Thus adding such checks to the form becomes very simple rather than using jQuery for it. Similarly it was important to add validation to the Register page.

 <input type="email" name="email" class="form-control" id="email" 
 placeholder="Email" data-remote="{{ url_for('admin.check_duplicate_email') }}"
 data-remote-error="Email Address already exists" required="">

This is the validation for the email field. Apart from checking whether the text entered by the user is an email id or not it was also important to check whether the email entered by the user exists in the database or not.

data-remote="{{ url_for('admin.check_duplicate_email')

This line calls a view function to check whether the email entered by the user is duplicate or unique. Here is the function

2.png

This function takes the email value from the request.args and then performs a simple check in the db to check for duplicate email. If the user doesnt exist then the validator receives a simple string “200 OK”. If the error is 404 then it gives the error defined by,

data-remote-error="Email Address already exists"

However this error only for the data-remote part. If the error is something else then it is handled by ,

class="help-block with-errors"

Thus without the hassle of using Ajax and JQuery we can easily add validation to the forms using Bootstrap Validator.

Continue ReadingAdding Client Side validation to Login and Registration Forms

How to parse json assets with gson

So most of us have json assets in our app which we parse on runtime to get the data and use it accordingly but what I have seen is that most of the people create a JSONObject or JSONArray after reading the json into an inputstream but then handling it becomes difficult since we have to manually extract every entity in each array which makes it bound to a lot of errors. A better approach to using it is making use of gson : an open source library by Google to serialise and deserialise Java object to (and from) Json. It’s pretty easy to use and makes the development process easy. For those of you not still convinced on using gson, I’d like to demonstrate the code we had to write to without using gson and the one using gson as well.

So to start with lets see the json file we’ll be using. It’s the events.json file from the open event project.

{
  "events": [
    {
      "color": "#fdfdfd",
      "email": "dev@fossasia.org",
      "end_time": "2015-07-14T00:00:00",
      "id": 4,
      "latitude": 37.783839,
      "location_name": "Moscone centre",
      "logo": "http://mysecureshell.readthedocs.org/en/latest/_images/logo_redhat.png",
      "longitude": -122.400546,
      "name": "FOSSASIA",
      "slogan": "Fossasia",
      "start_time": "2015-05-28T13:00:00",
      "url": "www.google.com"
    }
  ]
}

As you can see it has an object that has an array of event objects. So what we’ll first do is that we’ll get the whole json as a string by openeing an inputstream and then directing it to a buffer. Then we convert the buffer array to a string object.

String json = null;
try {
    InputStream inputStream = getAssets().open("events.json");
    int size = inputStream.available();
    byte[] buffer = new byte[size];
    inputStream.read(buffer);
    inputStream.close();
    json = new String(buffer, "UTF-8");

} catch (IOException e) {
    e.printStackTrace();
}

Now we have the json as a string which we can now parse it using a combination of JSONObject and JSONArray. First we’ll access data in the outer json object i.e. “events”. That’ll be done by

JSONObject jsonObject = new JSONObject(json);
JSONArray events = jsonObject.getJSONArray("events");

Now that we have the array, we can traverse it to get the objects inside events array

for (int j=0; j < events.length(); j++){
    JSONObject cit = events.getJSONObject(j);
    String color = jsonObject.getString("color");
    String email = jsonObject.getString("email"); 
    String endTime = jsonObject.getString("end_time");
    String id = jsonObject.getString("id");
    String latitude = jsonObject.getString("latitude");
    String locationName = jsonObject.getString("location_name");
    String logo = jsonObject.getString("logo");
    String longitude = jsonObject.getString("longitude");
    String name = jsonObject.getString("name");
    String slogan = jsonObject.getString("slogan");
    String startTime = jsonObject.getString("start_time");
    String url = jsonObject.getString("url");
}

This is how we go about it. Now for the exiting part.

We already have an Event data class which has the constructor, getters and setters etc.

public class Event {

    int id;

    String name;

    String email;

    String color;

    String logo;

    @SerializedName("start_time")
    String start;

    @SerializedName("end_time")
    String end;

    float latitude;

    float longitude;

    @SerializedName("location_name")
    String locationName;

    String url;

    String slogan;

    public Event(int id, String name, String email, String color, String logo, String start,
                 String end, float latitude, float longitude, String locationName, String url, String slogan) {
        this.id = id;
        this.name = name;
        this.email = email;
        this.color = color;
        this.logo = logo;
        this.start = start;
        this.end = end;
        this.latitude = latitude;
        this.longitude = longitude;
        this.locationName = locationName;
        this.url = url;
        this.slogan = slogan;
    }

    public String getEmail() {
        return email;
    }

    public void setEmail(String email) {
        this.email = email;
    }

    public String getColor() {
        return color;
    }

    public void setColor(String color) {
        this.color = color;
    }

    public String getUrl() {
        return url;
    }

    public void setUrl(String url) {
        this.url = url;
    }

    public String getSlogan() {
        return slogan;
    }

    public void setSlogan(String slogan) {
        this.slogan = slogan;
    }

    public int getId() {

        return id;

    }

    public void setId(int id) {
        this.id = id;
    }

    public String getName() {
        return name;
    }

    public void setName(String name) {
        this.name = name;
    }

    public String getLogo() {
        return logo;
    }

    public void setLogo(String logo) {
        this.logo = logo;
    }

    public String getStart() {
        return start;
    }

    public void setStart(String start) {
        this.start = start;
    }

    public String getEnd() {
        return end;
    }

    public void setEnd(String end) {
        this.end = end;
    }

    public float getLatitude() {
        return latitude;
    }

    public void setLatitude(float latitude) {
        this.latitude = latitude;
    }

    public float getLongitude() {
        return longitude;
    }

    public void setLongitude(float longitude) {
        this.longitude = longitude;
    }

    public String getLocationName() {
        return locationName;
    }

    public void setLocationName(String locationName) {
        this.locationName = locationName;
    }

}

Now here we name the parameters to the same as that in the json we have or we can just add @SerializedName(entity_name). Then we go to the code for actually retrieving the data from the json file using this data class. How we can do that is by first making a class that’ll get the array of events for us.

public class EventResponseList {
    @SerializedName("events")
    public List<Event> event;
}

Now all we do is

EventResponseList eventResponseList = gson.fromJson(json, EventResponseList.class);

We have a list of events that were in the json array.

Voila! That’s it. It’s so easy to get a list of all the events from the JSONArray and since the library is available for gradle, it’s even better. You can just add

compile 'com.google.code.gson:gson:2.7'

to your build.gradle dependencies and you’re good to go. Cheers!

Continue ReadingHow to parse json assets with gson
Read more about the article Update Fields with Array Input

Update Fields with Array Input

Screenshot from 2016-07-15 17:57:58.png

There are certain fields in form where instead of a single value, we need an array of values to be stored. This are fields under the same category but having multiple elements, e.g., Tracks, Session Types, Microlocations, Sponsors, Social Links and similar such fields. Now as we know the way of doing this using simple html is provide the <input> tag with property “name” as “<field_name>[]”. So suppose we want to save the names of the mutliple tracks, we will have something like this
Screenshot from 2016-07-15 18:02:09.png

But the problem begins when you want to update name of a particular element (in this case Track). How to do it? Should we delete and create entries again? That doesn’t sound too good, does it? So what should we do? Let’s see….

Continue ReadingUpdate Fields with Array Input

Ideas on using Celery with Flask for background tasks

Simply put, Celery is a background task runner. It can run time-intensive tasks in the background so that your application can focus on the stuff that matters the most. In context of a Flask application, the stuff that matters the most is listening to HTTP requests and returning response.

By default, Flask runs on a single-thread. Now if a request is executed that takes several seconds to run, then it will block all other incoming requests as it is single-threaded. This will be a very bad-experience for the user who is using the product. So here we can use Celery to move time-hogging part of that request to the background.

I would like to let you know that by “background”, Celery means another process. Celery starts worker processes for the running application and these workers receive work from the main application. Celery requires a broker to be used. Broker is nothing but a database that stores results of a celery task and provides a shared interface between main process and worker processes. The output of the work done by the workers is stored in the Broker. The main application can then access these results from the Broker.

Using Celery to set background tasks in your application is as simple as follows –

@celery.task
def background_task(*args, **kwargs):
    # do stuff
    # more stuff

Now the function background_task becomes function-able as a background task. To execute it as a background task, run –

task = background_task.delay(*args, **kwargs)
print task.state  # task current state (PENDING, SUCCESS, FAILURE)

Till now this may look nice and easy but it can cause lots of problems. This is because the background tasks run in different processes than the main application. So the state of the worker application differs from the real application.

One common problem because of this is the lack of request context. Since a celery task runs in a different process, so the request context is not available. Therefore the request headers, cookies and everything else is not available when the task actually runs. I too faced this problem and solved it using an excellent snippet I found on the Internet.

"""
Celery task wrapper to set request context vars and global
vars when a task is executed
Based on http://xion.io/post/code/celery-include-flask-request-context.html
"""
from celery import Task
from flask import has_request_context, make_response, request, g

from app import app  # the flask app


__all__ = ['RequestContextTask']


class RequestContextTask(Task):
    """Base class for tasks that originate from Flask request handlers
    and carry over most of the request context data.
    This has an advantage of being able to access all the usual information
    that the HTTP request has and use them within the task. Pontential
    use cases include e.g. formatting URLs for external use in emails sent
    by tasks.
    """
    abstract = True

    #: Name of the additional parameter passed to tasks
    #: that contains information about the original Flask request context.
    CONTEXT_ARG_NAME = '_flask_request_context'
    GLOBALS_ARG_NAME = '_flask_global_proxy'
    GLOBAL_KEYS = ['user']

    def __call__(self, *args, **kwargs):
        """Execute task code with given arguments."""
        call = lambda: super(RequestContextTask, self).__call__(*args, **kwargs)

        # set context
        context = kwargs.pop(self.CONTEXT_ARG_NAME, None)
        gl = kwargs.pop(self.GLOBALS_ARG_NAME, {})

        if context is None or has_request_context():
            return call()

        with app.test_request_context(**context):
            # set globals
            for i in gl:
                setattr(g, i, gl[i])
            # call
            result = call()
            # process a fake "Response" so that
            # ``@after_request`` hooks are executed
            # app.process_response(make_response(result or ''))

        return result

    def apply_async(self, args=None, kwargs=None, **rest):
        self._include_request_context(kwargs)
        self._include_global(kwargs)
        return super(RequestContextTask, self).apply_async(args, kwargs, **rest)

    def apply(self, args=None, kwargs=None, **rest):
        self._include_request_context(kwargs)
        self._include_global(kwargs)
        return super(RequestContextTask, self).apply(args, kwargs, **rest)

    def retry(self, args=None, kwargs=None, **rest):
        self._include_request_context(kwargs)
        self._include_global(kwargs)
        return super(RequestContextTask, self).retry(args, kwargs, **rest)

    def _include_request_context(self, kwargs):
        """Includes all the information about current Flask request context
        as an additional argument to the task.
        """
        if not has_request_context():
            return

        # keys correspond to arguments of :meth:`Flask.test_request_context`
        context = {
            'path': request.path,
            'base_url': request.url_root,
            'method': request.method,
            'headers': dict(request.headers),
        }
        if '?' in request.url:
            context['query_string'] = request.url[(request.url.find('?') + 1):]

        kwargs[self.CONTEXT_ARG_NAME] = context

    def _include_global(self, kwargs):
        d = {}
        for z in self.GLOBAL_KEYS:
            if hasattr(g, z):
                d[z] = getattr(g, z)
        kwargs[self.GLOBALS_ARG_NAME] = d

To run a task in Request context mode, do –

@celery.task(base=RequestContextTask, bind=True)
def background_task(self, *args, **kwargs):
    # do stuff
    # more stuff

If you are wondering what the RequestContextTask class does, it simply stores all request context vars and global vars when a background task is called (task.delay()) and then unpacks those values to their proper places when the task is about to be run. The above snippet can be easily extended to store any value.

Another challenge that some people may face is the occasional Parsing/Serialization error. This happens because the data being sent to/from a function that is to be background executed is too complex.

Serialization is the process of converting complex data structures and objects into a plain string. Serialization of data is necessary because the background tasks and the main thread run in different processes. Now think how will the main thread communicate the celery thread to do some task. This is done using serialization of the concerned data. So to avoid serialization errors, it is recommended that you make background tasks such that they require only simple arguments to run and they return only simple data.

So basically keeping small and simple tasks is recommended when using Celery. Follow this golden rule and you will not run into any problems.

 

{{ Repost from my personal blog | http://aviaryan.in/blog/gsoc/celery-flask-good-ideas.html }}

Continue ReadingIdeas on using Celery with Flask for background tasks

Using compression middleware in Node/Express

If you’re using ExpressJS to host your server, there’s a small performance tweak that you can do (works on universally on any kind of server – HTML templates to JSON REST APIs to image servers) – which is to enable GZIP compression.

The changed required to do it should be very simple. First install the compression npm package

 npm install --save compression

In you app.js (or wherever you start your express server), modify the code to include the compression middleware like this

var express = require('express);
var app = express();

var compression = require('compression');

app.use(compression());

// Rest of your app code
// app.get (endpoint, callback) 
// app.listen(port, callback)

This should easily reduce page loads time to the order of 15-20%.

For example, page load before compression middleware added 72f7abf2-4899-11e6-9cd5-68a7addaf3a6

 

And Page load time after compression is added69e6fba8-4899-11e6-9626-a25c26ea7d2b

Continue ReadingUsing compression middleware in Node/Express

Permission Decorators

A follow-up to one of my previous posts: Organizer Server Permissions System.

I recently had a requirement to create permission decorators for use in our REST APIs. There had to be separate decorators for Event and Services.

Event Permission Decorators

Understanding Event permissions is simple: Any user can create an event. But access to an event is restricted to users that have Event specific Roles (e.g. Organizer, Co-organizer, etc) for that event. The creator of an event is its Organizer, so he immediately gets access to that event. You can read about these roles in the aforementioned post.

So for Events, create operation does not require any permissions, but read/update/delete operations needed a decorator. This decorator would restrict access to users with event roles.

def can_access(func):
    """Check if User can Read/Update/Delete an Event.
    This is done by checking if the User has a Role in an Event.
    """
    @wraps(func)
    def wrapper(*args, **kwargs):
        user = UserModel.query.get(login.current_user.id)
        event_id = kwargs.get('event_id')
        if not event_id:
            raise ServerError()
        # Check if event exists
        get_object_or_404(EventModel, event_id)
        if user.has_role(event_id):
            return func(*args, **kwargs)
        else:
            raise PermissionDeniedError()
    return wrapper

The has_role(event_id) method of the User class determines if the user has a Role in an event.

# User Model class

    def has_role(self, event_id):
        """Checks if user has any of the Roles at an Event.
        """
        uer = UsersEventsRoles.query.filter_by(user=self, event_id=event_id).first()
        if uer is None:
            return False
        else:
            return True

Reading one particular event (/events/:id [GET]) can be restricted to users, but a GET request to fetch all the events (/events [GET]) should only be available to staff (Admin and Super Admin). So a separate decorator to restrict access to Staff members was needed.

def staff_only(func):
    @wraps(func)
    def wrapper(*args, **kwargs):
        user = UserModel.query.get(login.current_user.id)
        if user.is_staff:
            return func(*args, **kwargs)
        else:
            raise PermissionDeniedError()
    return wrapper

Service Permission Decorators

Service Permissions for a user are defined using Event Roles. What Role a user has in an Event determines what Services he has access to in that Event. Access here means permission to Create, Read, Update and Delete services. The User model class has four methods to determine the permissions for a Service in an event.

user.can_create(service, event_id)
user.can_read(service, event_id)
user.can_update(service, event_id)
user.can_delete(service, event_id)

So four decorators were needed to put alongside POST, GET, PUT and DELETE method handlers. I’ve pasted snippet for the can_update decorator. The rest are similar but with their respective permission methods for User class object.

def can_update(DAO):
    def decorator(func):
        @wraps(func)
        def wrapper(*args, **kwargs):
            user = UserModel.query.get(login.current_user.id)
            event_id = kwargs.get('event_id')
            if not event_id:
                raise ServerError()
            # Check if event exists
            get_object_or_404(EventModel, event_id)
            service_class = DAO.model
            if user.can_update(service_class, event_id):
                return func(*args, **kwargs)
            else:
                raise PermissionDeniedError()
        return wrapper
    return decorator

This decorator is a little different than can_access event decorator in a way that it takes an argument, DAO. DAO is Data Access Object. A DAO includes a database Model and methods to create, read, update and delete object of that model. The db model for a DAO would be the Service class for the object. You can look that the model class is taken from the DAO and used as the service class.

The can_create, can_read and can_delete decorators look exactly the same except they use their (obvious) permission methods on the User class object.

Continue ReadingPermission Decorators

Shell hacks

Custom Shell

Working with database models needs a lot of use of the flask shell. You can access it with:

python manage.py shell

The default shell is quite unintuitive. It doesn’t pretty print the outputs and has no support for auto-completion. I’ve installed IPython that takes care of that. But still working with models means writing a lot of import statements. Plus if there was some change in the code related to the app, then the shell had to restarted again so the changes could be loaded. Meaning writing the import statements again.

We were using Flask-Script and I wanted to run a custom shell that imports all the required modules, models and helper functions, so I don’t have to write them over and over. Some of them were as long as:

from open_event.models.users_events_roles import UsersEventsRoles

So I created a custom shell command with different context that overrides the default shell provided by the Flask-Script Manager. It was pretty easy with Flask-Script. One thing I had to keep in mind is that it needed to be in a different file than manage.py. Since manage.py was committed to source repo, changes to it would be tracked. So I needed a different file that could be excluded from the source repo. I created an smg.py that imported the Manager from the open_event module and overrides the shell command.

from flask_script import Shell

from open_event import manager
from open_event.models.user import User
from open_event.models.event import Event
from open_event.helpers.data import save_to_db, delete_from_db

def _make_context():
    return dict(
        uq=User.query,
        eq=Event.query,
        su=User.query.get(1),
        User=User,
        Event=Event,
        savetodb=save_to_db,
        deletefromdb=delete_from_db
    )


if __name__ == "__main__":
    manager.add_command('shell', Shell(make_context=_make_context))
    manager.run()

Place this smg.py file in the same directory as manage.py, so you can access it with python smg.py shell.

The code is pretty simple to understand. We import the Shell class from flask_script, create its object with our context and then add it to the manager as a command. _make_context contains what I usually like to have in my shell. It must always return a dictionary. The keys of this dictionary would be available as statements inside the shell with their values specified here.

This helps a lot. Most of the time I would be working with the super_admin user, and I would need its User object from time to time. The super_admin user is always going to be the first user (User object with id 1). So instead of from open_event.models.user import User; su = User.query.get(1) I could just use the su variable. Models like User and Event are also readily available (so are their base queries). This is the default context that I always keep, but many times you need more models than the ones specified here. Like when I was working with the permissions system.

from open_event.models.users_events_roles import UsersEventsRoles
from open_event.models.service import Service
from open_event.models.role import Role

def _make_context():
    return dict(
        # usual stuff
        UER=UsersEventsRoles,
        Service=Service,
        Role=Role
    )

You can even write a script that fetches all the database models (instance of sqlalchemy Model class) and then add them to the _make_context dictionary. I like to keep it minimum and differently named, so there are no conflicts of classes when try to auto-complete.

One more thing, you need to exclude the smg.py file so that git doesn’t track it. You can simply add it in the .git/info/exclude file.

I also wrote some useful one-liner bash commands.

Revision History

Every time someone updates the database models, he needs to migrate and provide the migrations file to others by committing it to the source. These files help us upgrade our already defined database. We work with Alembic. Regarding alembic revisions (migration files) you can keep two things in mind. One is the Head, that keeps track of the latest revision, and another is Current, that specifies what revision your tables in the database are based on. If your current revision is not a head, it means your database tables are not up-to-date. You need to upgrade (python manage.py db upgrade). The can be multiple heads in the revision history. python manage.py heads displays all of them. The current revision can be fetched with python manage.py current. I wanted something that automatically checks if current is at a head.

python manage.py db history | grep --color "$(python manage.py db current 2> /dev/null)|$(python manage.py db heads)|$"

You can specify it as an alias in .bashrc. This command displays the revision history with the head and current revision being colored.

Screenshot from 2016-07-11 16:34:18

You can identify the head as it is always appended by “(head)”. The other colored revision would be the current.

You can also reverse the output, by printing bottom to up. This is helpful when the revision history gets large and you have to scroll up to see the head.

alias rev='python manage.py db history | tac | grep --color "$(python manage.py db current 2> /dev/null)|$(python manage.py db heads)|$"'

Screenshot from 2016-07-11 16:42:30.png

Current Revision changer

The current revision is maintained at the database in the one-column alembic_version table. If for some reason the migration file for current revision no longer exists (like when changing to a branch at git that doesn’t have the file), alembic would raise an error. So to change the revision at the database I wrote a bash function.

change_db_current () {
    echo "Current Revision: "
    read current_rev
    echo "New Revision: "
    read new_rev

    echo 'c test \ update alembic_version set version_num='"'"$new_rev"'"' where version_num='"'"$current_rev"'"';' | sudo su - postgres -c psql
}

Enter the current revision hash (displayed in the alembic error) and the revision you need to point the current to, and it will update the table with the new revision. Sick right! It’s pretty stupid actually. It was when I didn’t know how to stamp in alembic. Anyways, it was useful some time ago.

Continue ReadingShell hacks

Writing tests for Open-Event

As our application and code base increased it became necessary to write tests for each functionality. Earlier we had tests only for basic functionalities like creating an event, editing an event, but then it is very important and also beneficial if we have tests for each and every small functionality. Hence we started writing proper tests. We divivded the tests into three folder

  • API
  • Functionality
  • Views

All the API related tests were in the above one whereas the basic functionalities were in the second one. The last folder was further divided into three parts

  • Admin Tests
  • Super-Admin Tests
  • Guest Pages

We had to test each and every functionality. For example let us look at the test file for the events. It looks like this:

class TestEvents(OpenEventViewTestCase):
    def test_events_list(self):
        with app.test_request_context():
            url = url_for('events.index_view')
            rv = self.app.get(url, follow_redirects=True)

            self.assertTrue("Manage Events" in rv.data, msg=rv.data)

    def test_events_create(self):
        with app.test_request_context():
            url = url_for('events.create_view')
            rv = self.app.get(url, follow_redirects=True)
            self.assertTrue("Create Event" in rv.data, msg=rv.data)

    def test_events_create_post(self):
        with app.test_request_context():
            custom_forms = ObjectMother.get_custom_form()
            url = url_for('events.create_view')
            data = POST_EVENT_DATA.copy()
            del data['copyright']
            data['start_date'] = '07/04/2016'
            data['start_time'] = '19:00'
            data['end_date'] = '07/04/2016'
            data['end_time'] = '22:00'
            data['custom_form[name]'] = ['session_form', 'speaker_form']
            data['custom_form[value]'] = [custom_forms.session_form, custom_forms.speaker_form]
            rv = self.app.post(url, follow_redirects=True, buffered=True, content_type='multipart/form-data',
                               data=data)
            self.assertTrue(POST_EVENT_DATA['name'] in rv.data, msg=rv.data)

    def test_events_create_post_publish(self):
        with app.test_request_context():
            url = url_for('events.create_view')
            data = POST_EVENT_DATA.copy()
            del data['copyright']
            data['start_date'] = '07/04/2016'
            data['start_time'] = '19:00'
            data['end_date'] = '07/04/2016'
            data['end_time'] = '22:00'
            data['state'] = 'Published'
            rv = self.app.post(url, follow_redirects=True, buffered=True, content_type='multipart/form-data',
                               data=data)
            self.assertTrue('unpublish' in rv.data, msg=rv.data)

    def test_events_create_post_publish_without_location_attempt(self):
        with app.test_request_context():
            custom_forms = ObjectMother.get_custom_form()
            url = url_for('events.create_view')
            data = POST_EVENT_DATA.copy()
            del data['copyright']
            data['start_date'] = '07/04/2016'
            data['start_time'] = '19:00'
            data['end_date'] = '07/04/2016'
            data['end_time'] = '22:00'
            data['location_name'] = ''
            data['state'] = u'Published'
            data['custom_form[name]'] = ['session_form', 'speaker_form']
            data['custom_form[value]'] = [custom_forms.session_form, custom_forms.speaker_form]
            rv = self.app.post(url, follow_redirects=True, buffered=True, content_type='multipart/form-data',
                               data=data)
            self.assertTrue('To publish your event please review the highlighted fields below' in rv.data, msg=rv.data)

    def test_events_edit(self):
        with app.test_request_context():
            event = ObjectMother.get_event()
            save_to_db(event, "Event saved")
            custom_forms = ObjectMother.get_custom_form(event.id)
            save_to_db(custom_forms, "Custom forms saved")
            url = url_for('events.edit_view', event_id=event.id)
            data = POST_EVENT_DATA.copy()
            del data['copyright']
            data['name'] = 'EditTestName'
            data['start_date'] = '07/04/2016'
            data['start_time'] = '19:00'
            data['end_date'] = '07/04/2016'
            data['end_time'] = '22:00'
            data['custom_form[name]'] = ['session_form', 'speaker_form']
            data['custom_form[value]'] = [custom_forms.session_form, custom_forms.speaker_form]
            rv = self.app.post(url, follow_redirects=True, buffered=True, content_type='multipart/form-data',
                               data=data)
            self.assertTrue('EditTestName' in rv.data, msg=rv.data)

    def test_event_view(self):
        with app.test_request_context():
            event = ObjectMother.get_event()
            save_to_db(event, "Event saved")
            url = url_for('events.details_view', event_id=event.id)
            rv = self.app.get(url, follow_redirects=True)
            self.assertTrue("event1" in rv.data, msg=rv.data)
            microlocation = ObjectMother.get_microlocation(event_id=event.id)
            track = ObjectMother.get_track(event_id=event.id)
            cfs = ObjectMother.get_cfs(event_id=event.id)
            save_to_db(track, "Track saved")
            save_to_db(microlocation, "Microlocation saved")
            save_to_db(cfs, "Call for speakers saved")
            rv = self.app.get(url, follow_redirects=True)
            self.assertTrue("event1" in rv.data, msg=rv.data)

    def test_event_publish(self):
        with app.test_request_context():
            event = ObjectMother.get_event()
            save_to_db(event, "Event saved")
            url = url_for('events.publish_event', event_id=event.id)
            rv = self.app.get(url, follow_redirects=True)
            event = DataGetter.get_event(event.id)
            self.assertEqual("Published", event.state, msg=event.state)

    def test_event_unpublish(self):
        with app.test_request_context():
            event = ObjectMother.get_event()
            event.state = "Published"
            save_to_db(event, "Event saved")
            url = url_for('events.unpublish_event', event_id=event.id)
            rv = self.app.get(url, follow_redirects=True)
            event = DataGetter.get_event(event.id)
            self.assertEqual("Draft", event.state, msg=event.state)

    def test_event_delete(self):
        with app.test_request_context():
            event = ObjectMother.get_event()
            save_to_db(event, "Event saved")
            url = url_for('events.trash_view', event_id=event.id)
            rv = self.app.get(url, follow_redirects=True)
            self.assertTrue("Your event has been deleted" in rv.data, msg=rv.data)

    def test_event_copy(self):
        with app.test_request_context():
            event = ObjectMother.get_event()
            save_to_db(event, "Event saved")
            url = url_for('events.copy_event', event_id=event.id)
            rv = self.app.get(url, follow_redirects=True)
            self.assertTrue("Copy of event1" in rv.data, msg=rv.data)

if __name__ == '__main__':
    unittest.main()

So this is the test file for the event part. As you can see we have tests for each and every small functionality

  1. test_events_list : Tests the list of events
  2. test_events_create: Tests whether the event creation page is displayed
  3. test_events_create_post: Tests whether the event is created on doing a POST
  4. test_events_create_post_publish : Tests whether the event is published on doing a POST through Publish button
  5. test_events_copy: Tests whether the event is copied properly or not

Thus each functionality related to an event is tested properly. Similarly not just for events but also for the other services like sessions:

import unittest

from tests.api.utils_post_data import POST_SESSION_DATA, POST_SPEAKER_DATA
from tests.object_mother import ObjectMother
from open_event import current_app as app
from open_event.helpers.data import save_to_db
from flask import url_for

from tests.views.view_test_case import OpenEventViewTestCase


class TestSessionApi(OpenEventViewTestCase):

    def test_sessions_list(self):
        with app.test_request_context():
            event = ObjectMother.get_event()
            save_to_db(event, "Event saved")
            session = ObjectMother.get_session(event.id)
            save_to_db(session, "Session Saved")
            url = url_for('event_sessions.index_view', event_id=event.id, session_id=session.id)
            rv = self.app.get(url, follow_redirects=True)
            self.assertTrue("Sessions" in rv.data, msg=rv.data)
            self.assertTrue("test" in rv.data, msg=rv.data)

    def test_session_create(self):
        with app.test_request_context():
            event = ObjectMother.get_event()
            save_to_db(event, "Event saved")
            custom_form = ObjectMother.get_custom_form(event.id)
            save_to_db(custom_form, "Custom form saved")
            url = url_for('event_sessions.create_view', event_id=event.id)
            rv = self.app.get(url, follow_redirects=True)
            self.assertTrue("Create Session" in rv.data, msg=rv.data)

    def test_session_create_post(self):
        with app.test_request_context():
            event = ObjectMother.get_event()
            save_to_db(event, "Event saved")
            custom_form = ObjectMother.get_custom_form(event.id)
            save_to_db(custom_form, "Custom form saved")
            data = POST_SESSION_DATA
            data.update(POST_SPEAKER_DATA)
            url = url_for('event_sessions.create_view', event_id=event.id)
            rv = self.app.post(url, follow_redirects=True, buffered=True, content_type='multipart/form-data', data=data)
            self.assertTrue(data['title'] in rv.data, msg=rv.data)

    def test_session_edit(self):
        with app.test_request_context():
            event = ObjectMother.get_event()
            save_to_db(event, "Event saved")
            custom_form = ObjectMother.get_custom_form(event.id)
            save_to_db(custom_form, "Custom form saved")
            session = ObjectMother.get_session(event.id)
            save_to_db(session, "Session saved")
            url = url_for('event_sessions.edit_view', event_id=event.id, session_id=session.id)
            rv = self.app.get(url, follow_redirects=True)
            self.assertTrue("Edit Session" in rv.data, msg=rv.data)

    def test_session_edit_post(self):
        with app.test_request_context():
            event = ObjectMother.get_event()
            save_to_db(event, "Event saved")
            custom_form = ObjectMother.get_custom_form(event.id)
            save_to_db(custom_form, "Custom form saved")
            session = ObjectMother.get_session(event.id)
            save_to_db(session, "Session saved")
            data = POST_SESSION_DATA
            data['title'] = 'TestSession2'
            url = url_for('event_sessions.edit_view', event_id=event.id, session_id=session.id)
            rv = self.app.post(url, follow_redirects=True, buffered=True, content_type='multipart/form-data', data=data)
            self.assertTrue("TestSession2" in rv.data, msg=rv.data)

    def test_session_accept(self):
        with app.test_request_context():
            session = ObjectMother.get_session()
            save_to_db(session, "Session Saved")
            url = url_for('event_sessions.accept_session', event_id=1, session_id=session.id)
            rv = self.app.get(url, follow_redirects=True)
            self.assertTrue("The session has been accepted" in rv.data, msg=rv.data)

    def test_session_reject(self):
        with app.test_request_context():
            session = ObjectMother.get_session()
            save_to_db(session, "Session Saved")
            url = url_for('event_sessions.reject_session', event_id=1, session_id=session.id)
            rv = self.app.get(url, follow_redirects=True)
            self.assertTrue("The session has been rejected" in rv.data, msg=rv.data)

    def test_session_delete(self):
        with app.test_request_context():
            session = ObjectMother.get_session()
            save_to_db(session, "Session Saved")
            url = url_for('event_sessions.delete_session', event_id=1, session_id=session.id)
            rv = self.app.get(url, follow_redirects=True)
            self.assertTrue("deleted" in rv.data, msg=rv.data)

    def test_session_view(self):
        with app.test_request_context():
            event = ObjectMother.get_event()
            save_to_db(event)
            session = ObjectMother.get_session()
            session.event_id = event.id
            save_to_db(session, "Session Saved")
            url = url_for('event_sessions.session_display_view', event_id=event.id, session_id=session.id)
            rv = self.app.get(url, follow_redirects=True)
            self.assertTrue("Short abstract" in rv.data, msg=rv.data)

    def test_wrong_form_config(self):
        with app.test_request_context():
            event = ObjectMother.get_event()
            save_to_db(event, "Event saved")
            url = url_for('event_sessions.create_view', event_id=event.id)
            rv = self.app.get(url, follow_redirects=True)
            self.assertFalse("incorrectly configured" in rv.data, msg=rv.data)

if __name__ == '__main__':
    unittest.main()

We see that there are tests for each functionality of the sessions. However these tests were simple to write. However there was problem in one aspect of writing tests. In the Event creation wizard there are steps where the sponsors, tracks, rooms are dynamically added to the event. How then should we test them. I wrote the test for the creation of sponsors in step -2

def test_events_create_post(self):
    with app.test_request_context():
        custom_forms = ObjectMother.get_custom_form()
        url = url_for('events.create_view')
        data = POST_EVENT_DATA.copy()
        del data['copyright']
        data['sponsors[name]'] = ['Sponsor 1', 'Sponsor 2']
        data['sponsors[type]'] = ['Gold', 'Silver']
        data['sponsors[url]'] = ["", ""]
        data['sponsors[description]'] = ["", ""]
        data['sponsors[level]'] = ["", ""]
        data['start_date'] = '07/04/2016'
        data['start_time'] = '19:00'
        data['end_date'] = '07/04/2016'
        data['end_time'] = '22:00'
        data['custom_form[name]'] = ['session_form', 'speaker_form']
        data['custom_form[value]'] = [custom_forms.session_form, custom_forms.speaker_form]
        data = ImmutableMultiDict(data)
        rv = self.app.post(url, follow_redirects=True, buffered=True, content_type='multipart/form-data',
                           data=data)
        self.assertTrue(POST_EVENT_DATA['name'] in rv.data, msg=rv.data)

        rv2 = self.app.get(url_for('events.details_view', event_id=1))
        self.assertTrue(data['sponsors[name]'] in rv2.data, msg=rv2.data)

Here on importing the data dict I dynamically add two sponsors to the dict. After that I convert the dict to an Immutablemulti-dict so that the multiple sponsors can be displayed. Then I pass this dict to the event creation view via a POST request and check whether the two sponsors are present in the details page or not.

Thus our test system is developed and improving. Still as we develop more functionalities we will write more tests 🙂

 

Continue ReadingWriting tests for Open-Event
Read more about the article How can you get an access to Instagram API?

How can you get an access to Instagram API?

unnamed.png

First of all you need to know that Instagram API uses OAuth 2.0 protocol. OAuth 2.0 provides a specific authorization flow for web apps, desktop apps and mobile apps. Instagram requires authentication before getting information from their API, but don’t be afraid it’s very simple.

Pre Requirements:

  • Created account in Instagram
  • Registered Client(You can create your own client here)

Instagram Developer Documentation.png

Requirements:

CLIENT_ID -79e1a142dbeabd57a3308c52ad43e31d
CLIENT_SECRET -34a6834081c44c20bd11e0a112a6adg1
REDIRECT_URI - http://127.0.0.1:8001/iCallback

You can get above information from https://www.instagram.com/developer/clients/manage/

CODE - You need to open page https://api.instagram.com/oauth/authorize/?client_id=CLIENT-ID&redirect_uri=REDIRECT-URI&response_type=code
https://api.instagram.com/oauth/authorize/?client_id=79e1a142dbeabd57a3308c52ad43e31d&redirect_uri=http://127.0.0.1:8001/iCallback&response_type=code

You will be redirected to

http://your-redirect-uri?code=CODE

In my case it looks like this:

http://127.0.0.1:8001/iCallback?code=2e122f3d76e8125b8b4982f16ed623c2

Now we have all information to get access token!

 curl -F 'client_id=CLIENT_ID'  
      -F 'client_secret=CLIENT_SECRET' 
      -F 'grant_type=authorization_code'  
      -F 'redirect_uri=REDIRECT_URI' 
      -F 'code=CODE'  https://api.instagram.com/oauth/access_token

if everything is ok you should receive

 { "access_token": "fb2e77d.47a0479900504cb3ab4a1f626d174d2d",
   "user": { "id": "1574083",
   "username": "rafal_kowalski",
   "full_name": "Rafal Kowalski",
   "profile_picture": "..." } } 

In Open Event we used it to get all media from instagram – to use it  as for example a background in an event details’ page

curl 'https://api.instagram.com/v1/users/self/media/recent/?access_token=ACCESS_TOKEN'

 

Continue ReadingHow can you get an access to Instagram API?