Testing Errors and Exceptions Using Unittest in Open Event Server

Like all other helper functions in FOSSASIA‘s Open Event Server, we also need to test the exception and error helper functions and classes. The error helper classes are mainly used to create error handler responses for known errors. For example we know error 403 is Access Forbidden, but we want to send a proper source message along with a proper error message to help identify and handle the error, hence we use the error classes. To ensure that future commits do not mismatch the error, we implemented the unit tests for errors.

There are mainly two kind of error classes, one are HTTP status errors and the other are the exceptions. Depending on the type of error we get in the try-except block for a particular API, we raise that particular exception or error.

Unit Test for Exception

Exceptions are written in this form:

@validates_schema
    def validate_quantity(self, data):
        if 'max_order' in data and 'min_order' in data:
            if data['max_order'] < data['min_order']:
                raise UnprocessableEntity({'pointer': '/data/attributes/max-order'},
                                          "max-order should be greater than min-order")

 

This error is raised wherever the data that is sent as POST or PATCH is unprocessable. For example, this is how we raise this error:

raise UnprocessableEntity({'pointer': '/data/attributes/min-quantity'},

           "min-quantity should be less than max-quantity")

This exception is raised due to error in validation of data where maximum quantity should be more than minimum quantity.

To test that the above line indeed raises an exception of UnprocessableEntity with status 422, we use the assertRaises() function. Following is the code:

 def test_exceptions(self):
        # Unprocessable Entity Exception
        with self.assertRaises(UnprocessableEntity):
            raise UnprocessableEntity({'pointer': '/data/attributes/min-quantity'},
                                      "min-quantity should be less than max-quantity")


In the above code,
with self.assertRaises() creates a context of exception type, so that when the next line raises an exception, it asserts that the exception that it was expecting is same as the exception raised and hence ensures that the correct exception is being raised

Unit Test for Error

In error helper classes, what we do is, for known HTTP status codes we return a response that is user readable and understandable. So this is how we raise an error:

ForbiddenError({'source': ''}, 'Super admin access is required')

This is basically the 403: Access Denied error. But with the “Super admin access is required” message it becomes far more clear. However we need to ensure that status code returned when this error message is shown still stays 403 and isn’t modified in future unwantedly.

Here, errors and exceptions work a little different. When we declare a custom error class, we don’t really raise that error. Instead we show that error as a response. So we can’t use the assertRaises() function. However what we can do is we can compare the status code and ensure that the error raised is the same as the expected one. So we do this:

def test_errors(self):
        with app.test_request_context():
            # Forbidden Error
            forbidden_error = ForbiddenError({'source': ''}, 'Super admin access is required')
            self.assertEqual(forbidden_error.status, 403)

            # Not Found Error
            not_found_error = NotFoundError({'source': ''}, 'Object not found.')
            self.assertEqual(not_found_error.status, 404)


Here we firstly create an object of the error class
ForbiddenError with a sample source and message. We then assert that the status attribute of this object is 403 which ensures that this error is of the Access Denied type using the assertEqual() function, which is what was expected.
The above helps us maintain that no one in future unknowingly or by mistake changes the error messages and status code so as to maintain the HTTP status codes in the response.


Resources:

Open Event Server: Testing Image Resize Using PIL and Unittest

FOSSASIA‘s Open Event Server project uses a certain set of functions in order to resize image from its original, example to thumbnail, icon or larger image. How do we test this resizing of images functions in Open Event Server project? To test image dimensions resizing functionality, we need to verify that the the resized image dimensions is same as the dimensions provided for resize.  For example, in this function, we provide the url for the image that we received and it creates a resized image and saves the resized version.

def create_save_resized_image(image_file, basewidth, maintain_aspect, height_size, upload_path,
                              ext='jpg', remove_after_upload=False, resize=True):
    """
    Create and Save the resized version of the background image
    :param resize:
    :param upload_path:
    :param ext:
    :param remove_after_upload:
    :param height_size:
    :param maintain_aspect:
    :param basewidth:
    :param image_file:
    :return:
    """
    filename = '{filename}.{ext}'.format(filename=get_file_name(), ext=ext)
    image_file = cStringIO.StringIO(urllib.urlopen(image_file).read())
    im = Image.open(image_file)

    # Convert to jpeg for lower file size.
    if im.format is not 'JPEG':
        img = im.convert('RGB')
    else:
        img = im

    if resize:
        if maintain_aspect:
            width_percent = (basewidth / float(img.size[0]))
            height_size = int((float(img.size[1]) * float(width_percent)))

        img = img.resize((basewidth, height_size), PIL.Image.ANTIALIAS)

    temp_file_relative_path = 'static/media/temp/' + generate_hash(str(image_file)) + get_file_name() + '.jpg'
    temp_file_path = app.config['BASE_DIR'] + '/' + temp_file_relative_path
    dir_path = temp_file_path.rsplit('/', 1)[0]

    # create dirs if not present
    if not os.path.isdir(dir_path):
        os.makedirs(dir_path)

    img.save(temp_file_path)
    upfile = UploadedFile(file_path=temp_file_path, filename=filename)

    if remove_after_upload:
        os.remove(image_file)

    uploaded_url = upload(upfile, upload_path)
    os.remove(temp_file_path)

    return uploaded_url


In this function, we send the
image url, the width and height to be resized to, and the aspect ratio as either True or False along with the folder to be saved. For this blog, we are gonna assume aspect ratio is False which means that we don’t maintain the aspect ratio while resizing. So, given the above mentioned as parameter, we get the url for the resized image that is saved.
To test whether it has been resized to correct dimensions, we use Pillow or as it is popularly know, PIL. So we write a separate function named getsizes() within which get the image file as a parameter. Then using the Image module of PIL, we open the file as a JpegImageFile object. The JpegImageFile object has an attribute size which returns (width, height). So from this function, we return the size attribute. Following is the code:

def getsizes(self, file):
        # get file size *and* image size (None if not known)
        im = Image.open(file)
        return im.size


As we have this function, it’s time to look into the unit testing function. So in unit testing we set dummy width and height that we want to resize to, set aspect ratio as false as discussed above. This helps us to test that both width and height are properly resized. We are using a creative commons licensed image for resizing. This is the code:

def test_create_save_resized_image(self):
        with app.test_request_context():
            image_url_test = 'https://cdn.pixabay.com/photo/2014/09/08/17/08/hot-air-balloons-439331_960_720.jpg'
            width = 500
            height = 200
            aspect_ratio = False
            upload_path = 'test'
            resized_image_url = create_save_resized_image(image_url_test, width, aspect_ratio, height, upload_path, ext='png')
            resized_image_file = app.config.get('BASE_DIR') + resized_image_url.split('/localhost')[1]
            resized_width, resized_height = self.getsizes(resized_image_file)


In the above code from
create_save_resized_image, we receive the url for the resized image. Since we have written all the unittests for local settings, we get a url with localhost as the server set. However, we don’t have the server running so we can’t acces the image through the url. So we build the absolute path to the image file from the url and store it in resized_image_file. Then we find the sizes of the image using the getsizes function that we have already written. This  gives us the width and height of the newly resized image. We make an assertion now to check whether the width that we wanted to resize to is equal to the actual width of the resized image. We make the same check with height as well. If both match, then the resizing function had worked perfectly. Here is the complete code:

def test_create_save_resized_image(self):
        with app.test_request_context():
            image_url_test = 'https://cdn.pixabay.com/photo/2014/09/08/17/08/hot-air-balloons-439331_960_720.jpg'
            width = 500
            height = 200
            aspect_ratio = False
            upload_path = 'test'
            resized_image_url = create_save_resized_image(image_url_test, width, aspect_ratio, height, upload_path, ext='png')
            resized_image_file = app.config.get('BASE_DIR') + resized_image_url.split('/localhost')[1]
            resized_width, resized_height = self.getsizes(resized_image_file)
            self.assertTrue(os.path.exists(resized_image_file))
            self.assertEqual(resized_width, width)
            self.assertEqual(resized_height, height)


In open event orga server, we use this resize function to basically create 3 resized images in various modules, such as events, users,etc. The 3 sizes are names – Large, Thumbnail and Icon. Depending on the one more suitable we use it avoiding the need to load a very big image for a very small div. The exact width and height for these 3 sizes can be changed from the admin settings of the project. We use the same technique as mentioned above. We run a loop to check the sizes for all these. Here is the code:

def test_create_save_image_sizes(self):
        with app.test_request_context():
            image_url_test = 'https://cdn.pixabay.com/photo/2014/09/08/17/08/hot-air-balloons-439331_960_720.jpg'
            image_sizes_type = "event"
            width_large = 1300
            width_thumbnail = 500
            width_icon = 75
            image_sizes = create_save_image_sizes(image_url_test, image_sizes_type)

            resized_image_url = image_sizes['original_image_url']
            resized_image_url_large = image_sizes['large_image_url']
            resized_image_url_thumbnail = image_sizes['thumbnail_image_url']
            resized_image_url_icon = image_sizes['icon_image_url']

            resized_image_file = app.config.get('BASE_DIR') + resized_image_url.split('/localhost')[1]
            resized_image_file_large = app.config.get('BASE_DIR') + resized_image_url_large.split('/localhost')[1]
            resized_image_file_thumbnail = app.config.get('BASE_DIR') + resized_image_url_thumbnail.split('/localhost')[1]
            resized_image_file_icon = app.config.get('BASE_DIR') + resized_image_url_icon.split('/localhost')[1]

            resized_width_large, _ = self.getsizes(resized_image_file_large)
            resized_width_thumbnail, _ = self.getsizes(resized_image_file_thumbnail)
            resized_width_icon, _ = self.getsizes(resized_image_file_icon)

            self.assertTrue(os.path.exists(resized_image_file))
            self.assertEqual(resized_width_large, width_large)
            self.assertEqual(resized_width_thumbnail, width_thumbnail)
            self.assertEqual(resized_width_icon, width_icon)

Resources:

Documenting Open Event API Using API-Blueprint

FOSSASIA‘s Open Event Server API documentation is done using an api-blueprint. The API Blueprint language is a format used to describe API in an API blueprint file, where a blueprint file (or a set of files) is such that describes an API using the API Blueprint language. To follow up with the blueprint, an apiary editor is used. This editor is responsible for rendering the API blueprint and printing the result in user readable API documented format. We create the API blueprint manually.

Using API Blueprint:-
We create the API blueprint by first adding the name and metadata for the API we aim to design. This step looks like this :-

FORMAT: V1
HOST: https://api.eventyay.com

# Open Event API Server

The Open Event API Server

# Group Authentication

The API uses JWT Authentication to authenticate users to the server. For authentication, you need to be a registered user. Once you have registered yourself as an user, you can send a request to get the access_token.This access_token you need to then use in Authorization header while sending a request in the following manner: `Authorization: JWT <access_token>`


API blueprint starts with the metadata, here FORMAT and HOST are defined metadata. FORMAT keyword specifies the version of API Blueprint . HOST defines the host for the API.

The heading starts with # and the first heading is regarded as the name of the API.

NOTE – Also all the heading starts with one or more # symbol. Each symbol indicates the level of the heading. One # symbol followed by heading serves as the top level i.e. one # = Top Level. Similarly for  ## = second level and so on. This is in compliance with normal markdown format.
        Following the heading section comes the description of the API. Further, headings are used to break up the description section.

Resource Groups:
—————————–
    By using group keyword at the starting of a heading , we create a group of related resources. Just like in below screenshot we have created a Group Users.

# Group Users

For using the API you need(mostly) to register as an user. Registering gives you access to all non admin API endpoints. After registration, you need to create your JWT access token to send requests to the API endpoints.


| Parameter | Description | Type | Required |
|:----------|-------------|------|----------|
| `name`  | Name of the user | string | - |
| `password` | Password of the user | string | **yes** |
| `email` | Email of the user | string | **yes** |

 

Resources:
——————
    In the Group Users we have created a resource Users Collection. The heading specifies the URI used to access the resource inside of the square brackets after the heading. We have used here parameters for the resource URI which takes us into how to add parameters to the URI. Below code shows us how to add parameters to the resource URI.

## Users Collection [/v1/users{?page%5bsize%5d,page%5bnumber%5d,sort,filter}]
+ Parameters
    + page%5bsize%5d (optional, integer, `10`) - Maximum number of resources in a single paginated response.
    + page%5bnumber%5d (optional, integer, `2`) - Page number to fetchedfor the paginated response.
    + sort (optional, string, `email`) - Sort the resources according to the given attribute in ascending order. Append '-' to sort in descending order.
    + filter(optional, string, ``) - Filter according to the flask-rest-jsonapi filtering system. Please refer: http://flask-rest-jsonapi.readthedocs.io/en/latest/filtering.html for more.

 

Actions:
————–
    An action is specified with a sub-heading inside of  a resource as the name of Action, followed by HTTP method inside the square brackets.
    Before we get on further, let us discuss what a payload is. A payload is an HTTP transaction message including its discussion and any additional assets such as entity-body validation schema.

There are two payloads inside an Action:

  1. Request: It is a payload containing one specific HTTP Request, with Headers and an optional body.
  2. Response: It is a payload containing one HTTP Response.

A payload may have an identifier-a string for a request payload or an HTTP status code for a response payload.

+ Request

    + Headers

            Accept: application/vnd.api+json

            Authorization: JWT <Auth Key>

+ Response 200 (application/vnd.api+json)


Types of HTTP methods for Actions:

  • GET – In this action, we simply send the header data like Accept and Authorization and no body. Along with it we can send some GET parameters like page[size]. There are two cases for GET: List and Detail. For example, if we consider users, a GET for List helps us retrieve information about all users in the response, while Details, helps us retrieve information about a particular user.

The API Blueprint examples implementation of both GET list and detail request and response are as follows.

### List All Users [GET]
Get a list of Users.

+ Request

    + Headers

            Accept: application/vnd.api+json

            Authorization: JWT <Auth Key>

+ Response 200 (application/vnd.api+json)

        {
            "meta": {
                "count": 2
            },
            "data": [
                {
                    "attributes": {
                        "is-admin": true,
                        "last-name": null,
                        "instagram-url": null,

 

### Get Details [GET]
Get a single user.

+ Request

    + Headers

            Accept: application/vnd.api+json

            Authorization: JWT <Auth Key>

+ Response 200 (application/vnd.api+json)

        {
            "data": {
                "attributes": {
                    "is-admin": false,
                    "last-name": "Doe",
                    "instagram-url": "http://instagram.com/instagram",

 

  • POST – In this action, apart from the header information, we also need to send a data. The data must be correct with jsonapi specifications. A POST body data for an users API would look something like this:
### Create User [POST]
Create a new user using an email, password and an optional name.

+ Request (application/vnd.api+json)

    + Headers

            Authorization: JWT <Auth Key>

    + Body

            {
              "data":
              {
                "attributes":
                {
                  "email": "[email protected]",
                  "password": "password",


A POST request with this data, would create a new entry in the table and then return in jsonapi format the particular entry that was made into the table along with the id assigned to this new entry.

  • PATCH – In this action, we change or update an already existing entry in the database. So It has a header data like all other requests and a body data which is almost similar to POST except that it also needs to mention the id of the entry that we are trying to modify.
### Update User [PATCH]
+ `id` (integer) - ID of the record to update **(required)**

Update a single user by setting the email, email and/or name.

Authorized user should be same as user in request body or must be admin.

+ Request (application/vnd.api+json)

    + Headers

            Authorization: JWT <Auth Key>

    + Body

            {
              "data": {
                "attributes": {
                  "password": "password1",
                  "avatar_url": "http://example1.com/example1.png",
                  "first-name": "Jane",
                  "last-name": "Dough",
                  "details": "example1",
                  "contact": "example1",
                  "facebook-url": "http://facebook.com/facebook1",
                  "twitter-url": "http://twitter.com/twitter1",
                  "instagram-url": "http://instagram.com/instagram1",
                  "google-plus-url": "http://plus.google.com/plus.google1",
                  "thumbnail-image-url": "http://example1.com/example1.png",
                  "small-image-url": "http://example1.com/example1.png",
                  "icon-image-url": "http://example1.com/example1.png"
                },
                "type": "user",
                "id": "2"
              }
            }

Just like in POST, after we have updated our entry, we get back as response the new updated entry in the database.

  • DELETE – In this action, we delete an entry from the database. The entry in our case is soft deleted by default. Which means that instead of deleting it from the database, we set the deleted_at column with the time of deletion. For deleting we just need to send header data and send a DELETE request to the proper endpoint. If deleted successfully, we get a response as “Object successfully deleted”.
### Delete User [DELETE]
Delete a single user.

+ Request

    + Headers

            Accept: application/vnd.api+json

            Authorization: JWT <Auth Key>

+ Response 200 (application/vnd.api+json)

        {
          "meta": {
            "message": "Object successfully deleted"
          },
          "jsonapi": {
            "version": "1.0"
          }
        }


How to check after manually entering all these? We can use the
apiary website to render it, or simply use different renderer to do it. How? Checkout for my next blog on apiary and aglio.

Learn more about api blueprint here: https://apiblueprint.org/

Open Event Server: Working with Migration Files

FOSSASIA‘s Open Event Server uses alembic migration files to handle all database operations and updations.  From creating tables to updating tables and database, all works with help of the migration files.
However, many a times we tend to miss out that automatically generated migration files mainly drops and adds columns rather than just changing them. One example of this would be:

def upgrade():
    ### commands auto generated by Alembic - please adjust! ###
    op.add_column('session', sa.Column('submission_date', sa.DateTime(), nullable=True))
    op.drop_column('session', 'date_of_submission')

Here, the idea was to change the has_session_speakers(string) to is_session_speakers_enabled (boolean), which resulted in the whole dropping of the column and creation of a new boolean column. We realize that, on doing so we have the whole data under  has_session_speakers lost.

How to solve that? Here are two ways we can follow up:

  • op.alter_column:
    ———————————-

When update is as simple as changing the column names, then we can use this. As discussed above, usually if we migrate directly after changing a column in our model, then the automatic migration created would drop the old column and create a new column with the changes. But on doing this in the production will cause huge loss of data which we don’t want. Suppose we want to just change the name of the column of start_time to starts_at. We don’t want the entire column to be dropped. So an alternative to this is using op.alter_column. The two main necessary parameters of the op.alter_column is the table name and the column which you are willing to alter. The other parameters include the new changes. Some of the commonly used parameters are:

  1. nullable Optional: specify True or False to alter the column’s nullability.
  2. new_column_name – Optional; specify a string name here to indicate the new name within a column rename operation.
  3. type_Optional: a TypeEngine type object to specify a change to the column’s type. For SQLAlchemy types that also indicate a constraint (i.e. Boolean, Enum), the constraint is also generated.
  4. autoincrement –  Optional: set the AUTO_INCREMENT flag of the column; currently understood by the MySQL dialect.
  5. existing_typeOptional: a TypeEngine type object to specify the previous type. This is required for all column alter operations that don’t otherwise specify a new type, as well as for when nullability is being changed on a column.

    So, for example, if you want to change a column name from “start_time” to “starts_at” in events table you would write:
    op.alter_column(‘events’, ‘start_time’, new_column_name=’starts_at’)
def upgrade():
    ### commands auto generated by Alembic - please adjust! ###
    op.alter_column('sessions_version', 'end_time', new_column_name='ends_at')
    op.alter_column('sessions_version', 'start_time', new_column_name='starts_at')
    op.alter_column('events_version', 'end_time', new_column_name='ends_at')
    op.alter_column('events_version', 'start_time', new_column_name='starts_at')


Here,
session_version and events_version are the tables name altering columns start_time to starts_at and end_time to ends_at with the op_alter_column parameter new_column_name.

  • op.execute:
    ——————–

Now with alter_column, most of the alteration in the column name or constraints or types is achievable. But there can be a separate scenario for changing the column properties. Suppose I change a table with column “aspect_ratio” which was a string column and had values “on” and “off” and want to convert the type to Boolean True/False. Just changing the column type using alte_column() function won’t work since we need to also modify the whole data. So, sometimes we need to execute raw SQL commands. To do that, we can use the op.execute() function.
The way it is done:

def upgrade():
    ### commands auto generated by Alembic - please adjust! ###
    op.execute("ALTER TABLE image_sizes ALTER full_aspect TYPE boolean USING CASE 
            full_aspect WHEN 'on' THEN TRUE ELSE FALSE END", execution_options=None)

    op.execute("ALTER TABLE image_sizes ALTER icon_aspect TYPE boolean USING CASE 
            icon_aspect WHEN 'on' THEN TRUE ELSE FALSE END", execution_options=None)

    op.execute("ALTER TABLE image_sizes ALTER thumbnail_aspect TYPE boolean USING CASE 
            thumbnail_aspect WHEN 'on' THEN TRUE ELSE FALSE END"execution_options=None)

For a little more advanced use of op.execute() command will be:

op.alter_column('events', 'type', new_column_name='event_type_id')
    op.alter_column('events_version', 'type', new_column_name='event_type_id')
    op.execute('INSERT INTO event_types(name, slug) SELECT DISTINCT event_type_id, 
                lower(replace(regexp_replace(event_type_id, \'& |,\', \'\', \'g\'),
                \' \', \'-\')) FROM events where not exists (SELECT 1 FROM event_types 
                where event_types.name=events.event_type_id) and event_type_id is not
                null;')
    op.execute('UPDATE events SET event_type_id = (SELECT id FROM event_types WHERE 
                event_types.name=events.event_type_id)')
    op.execute('ALTER TABLE events ALTER COLUMN event_type_id TYPE integer USING 
                event_type_id::integer')

In this example:

  • op.alter_column() renames the column type to event_type_id of events table
  • op.execute() does the following:
  • Inserts into column name of event_types table the value of event_type_idN (which previously contained the name of the event_type) from events table, and
  • Inserts into slug column of event_types table the value of event_type_id where all letters are changed to lowercase; “& ” and “,” to “”; and spaces to “-”.
    1. Checks whether a type with that name already exists so as to disallow any duplicate entries in the event_types table.
    2. Checks whether the event_type_id is null because name of event_types table cannot be null.

You can learn more on Alembic migrations here: http://alembic.zzzcomputing.com/en/latest/ops.html

Handling date-time in the Open Event Project

Handling date time in code becomes little tricky when the project is used internationally because then there comes additional term Timezone. Timezone is a property of a location which needs to be considered while comparing that time with the time in another location. For example – there are two villages A and B. One day Ram from village A calls his friend Shyam in village B at 8:00 am to wish “good morning”. But Shyam receives Ram’s call at 6pm on same day and he replies “good evening”. That means village A’s timezone is 10 hrs  behind village B’s timezone. So here we need some reference timezone about which all other timezones can be declared. This is where UTC (Coordinated Universal Time) comes into play. UTC is reference timezone by which all the timezones are declared. For example – Indian timezone is 5 hrs and 30 mins ahead of UTC which is denoted as UTC+05:30. In languages, these timezones are declared in date time library using constants such as ‘Asia/Kolkata’ which is Indian Standard Time. I will be talking about working with  date time in python in this blog. In the FOSSASIA’s Open Event project since it is event management system, handling date-time with the timezone is one of the important tasks.

Here is the relevant code:

>>> import datetime
>>> import pytz
>>> now = datetime.datetime.now()

datetime.datetime.now()  returns naive datetime as of the time setting of the machine on which the code is running. Naive date means it doesn’t contain any info about the timezone. It just contains some number values of year, month, hours etc. So just by looking at naive date we cannot actually understand the time. There comes aware datetime which contains timezone info.

>> now
datetime.datetime(2017, 5, 12, 21, 46, 16, 909983)
>>> now.isoformat()
'2017-05-12T21:46:16.909983'
>>> aware_now = pytz.timezone('Asia/Kolkata').localize(now)
>>> aware_now
datetime.datetime(2017, 5, 12, 21, 46, 16, 909983, tzinfo=<DstTzInfo 'Asia/Kolkata' IST+5:30:00 STD>

Pytz provides timezone object which takes string argument for timezone which has localize method which adds timezone info to the datetime object. Hence now aware datetime has timezone info too. Now if we print the time.

>>> aware_now.isoformat()
'2017-05-12T21:46:16.909983+05:30

We get the +05:30 extra string at the end which gives timezone info. +05:30 means the timezone is 5 hrs and 30 mins ahead of UTC timezone. The comparison between datetimes can be made between naive-naive and aware-aware. If we try to compare between naive and aware,

>>> now < aware_now
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: can't compare offset-naive and offset-aware datetimes

>>> now2 = datetime.datetime.now()
>>> now2
datetime.datetime(2017, 5, 15, 9, 44, 25, 990666)
>>> now < now2
True
>>> aware_now.tzinfo
<DstTzInfo 'Asia/Kolkata' IST+5:30:00 STD>

tzinfo carries timezone info of the datetime object. We can make aware date to unaware by method replacing tzinfo to None.

>>> unaware_now = aware_now.replace(tzinfo=None)
>>> unaware_now.isoformat()
'2017-05-12T21:46:16.909983'

Formating datetime is done by mostly these two methods. One of which takes string format in which the result is required and another returns datetime in iso format.

>>> now.strftime('%Y-%m-%dT%H:%M:%S%z')
'2017-05-12T21:46:16'
>>> aware_now.strftime('%Y-%m-%dT%H:%M:%S%z')
'2017-05-12T21:46:16+0530'

>>> now.time().isoformat()
'21:46:16.909983'
>>> now.date().isoformat()
'2017-05-12'

>>> now_in_brazil_east = datetime.datetime.now(pytz.timezone('Brazil/East'))
>>> now_in_brazil_east.isoformat()
'2017-05-15T06:49:51.311012-03:00'

We can pass timezone argument to the now method to get current time in the passed timezone. But the care must be taken as this will use the timezone setting of the machine on which code is running to calculate the time at the supplied timezone.

Application

In the FOSSASIA’s Open Event Project, date-time is taken from the user along with the timezone like in one of the example shown below.

Date Time Getter Open Event
Date Time Getter in Create Event Step 1 Open Event Front End

This is part of the event creation page where user has to provide date-time along with the timezone choice. At back-end the date-time and timezone are stored separately in the database. The event model looks like

class Event(db.Model):
 """Event object table"""
 ...
 start_time = db.Column(db.DateTime, nullable=False)
 end_time = db.Column(db.DateTime, nullable=False)
 timezone = db.Column(db.String, nullable=False, default="UTC")
 ...

The comparison between the stored date-time info with the real time cannot be done directly. Since the timezones of the both times need to be considered along with the date-time values while comparison. Likewise there is one case in the project code where tickets are filtered based on the start time.

def get_sales_open_tickets(event_id, event_timezone='UTC'):
  tickets = Ticket.query.filter(Ticket.event_id == event_id).filter(
      Ticket.sales_start <= datetime.datetime.now(pytz.timezone(event_timezone)).replace(tzinfo=None)).filter(
      Ticket.sales_end >= datetime.datetime.now(pytz.timezone(event_timezone)).replace(tzinfo=None))
  …

In this case, first current time is found out using timezone method in the timezone which is stored as a separate data field in the database. Since comparison cannot be done between aware and naive date-time. Hence once current date-time is found out in the user’s timezone, it is made naive using replace method which makes the aware date-time into naive again. Hence can be compared with the naive date-time stored already.

Sorting language-translation in Open Event Server project using Jinja 2 dictsort.

Working on the Open Event Server project an issue about arranging language-translation listing in alphabetical order came up. To solve this issue of language listing arrangement i.e. #2817, I found the ‘d0_dictsort’ function in jinja2 to sort dictionaries. It is a defined in jinja2.filters. Python dicts are unsorted and in our web application we at times may want to order them by either their key or value. So this function comes handy.

This is what the function looks like:

do_dictsort(value, case_sensitive=False, by='key')

We can write them in three ways as:

{% for record in my_dictionary|dictsort %}
    case insensitive and sort the dict by key

{% for record in my_dictionary|dicsort(true) %}
    case sensitive and sort the dict by key

{% for record in my_dictionary|dictsort(false, 'value') %}
    sort the dict by value, normally sorted and case insensitive
  1.       The first way is easily understood that dict has been sorted by key not taking case into consideration. It is just in the same way written as dictsort(false).
  2.       Second way is basically the first being case sensitive. dictsort(true) here tells us that case is sensitive.
  3.      Third way is dictsort(false,’value’). The first parameter defines that case insensitive while second parameter defines that it is sorted by ‘value’.

The issues was to sort translation selector for the page in alphabetical order. The languages were stored in a dictionary which to change in order, I found this function very easy and useful.

Basically what we had was:

This is how the function was used in the code for the sort. Like this:

<ul class="dropdown-menu lang-list">
   {% for code in all-languages|dictsort(false,'value') %}
       <li><a  href="#" style="#969191" class="translate" id="{{ code[0] }}">{{  all_languages[code[0]] }}<>a><li>
    {% endfor %}
<ul>


Here:
{{ all_languages }} is the list which contained the languages like French, English, etc., which could be accessed with its global language code. code here(index for all_languages) is a tuple of {‘global_language_code’,’language’} (An example would be (‘fr’,’French’), so code[0] gave me the language_code.

Finally, the result:

This is one of the simple ways to sort your dictionaries.

Open Event Server: No (no-wrap) Ellipsis using jquery!

Yes, the title says it all i.e., Enabling multiple line ellipsis. This was used to solve an issue to keep Session abstract view within 200 characters (#3059) on FOSSASIA‘s Open Event Server project.

There is this one way to ellipsis a paragraph in html-css and that is by using the text-overflow property:

.div_class{
white-space: nowrap;
overflow: hidden;
text-overflow: ellipsis;
}’’

But the downside of this is the one line ellipis. Eg: My name is Medozonuo. I am…..

And here you might pretty much want to ellipsis after a few characters in multiple lines, given that your div space is small and you do want to wrap your paragraph. Or maybe not.

So jquery to the rescue.

There are two ways you can easily do this multiple line ellipsis:

1) Height-Ellipsis (Using the do-while loop):

//script:
if ($('.div_class').height() > 100) {
    var words = $('.div_class').html().split(/\s+/);
    words.push('...');

    do {
        words.splice(-2, 1);
        $('.div_class').html( words.join(' ') );
    } while($('.div_class').height() > 100);
}

Here, you check for the div content’s height and split the paragraph after that certain height and add a “…”, do- while making sure that the paragraphs are in multiple lines and not in one single line. But checkout for that infinite loop.

2) Length-Ellipsis (Using substring function):  

//script:
$.each($('.div_class'), function() {
        if ($(this).html().length > 100) {
               var cropped_words = $(this).html();
               cropped_words = cropped_words.substring(0, 200) + "...";
               $(this).html(cropped_words);
        }
 });

Here, you check for the length/characters rather than the height, take in the substring of the content starting from 0-th character to the 200-th character and then add in extra “…”.

This is exactly how I used it in the code.

$.each($('.short_abstract',function() {
   if ($(this).html().length > 200) {
       var  words = $(this).html();
       words = words.substring(0,200 + "...";
       $(this).html(words);
    }
});


So ellipsing paragraphs over heights and lengths can be done using jQuery likewise.

DetachedInstanceError: Dealing with Celery, Flask’s app context and SQLAlchemy in the Open Event Server

In the open event server project, we had chosen to go with celery for async background tasks. From the official website,

What is celery?

Celery is an asynchronous task queue/job queue based on distributed message passing.

What are tasks?

The execution units, called tasks, are executed concurrently on a single or more worker servers using multiprocessing.

After the tasks had been set up, an error constantly came up whenever a task was called

The error was:

DetachedInstanceError: Instance <User at 0x7f358a4e9550> is not bound to a Session; attribute refresh operation cannot proceed

The above error usually occurs when you try to access the session object after it has been closed. It may have been closed by an explicit session.close() call or after committing the session with session.commit().

The celery tasks in question were performing some database operations. So the first thought was that maybe these operations might be causing the error. To test this theory, the celery task was changed to :

@celery.task(name='lorem.ipsum')
def lorem_ipsum():
    pass

But sadly, the error still remained. This proves that the celery task was just fine and the session was being closed whenever the celery task was called. The method in which the celery task was being called was of the following form:

def restore_session(session_id):
    session = DataGetter.get_session(session_id)
    session.deleted_at = None
    lorem_ipsum.delay()
    save_to_db(session, "Session restored from Trash")
    update_version(session.event_id, False, 'sessions_ver')


In our app, the app_context was not being passed whenever a celery task was initiated. Thus, the celery task, whenever called, closed the previous app_context eventually closing the session along with it. The solution to this error would be to follow the pattern as suggested on http://flask.pocoo.org/docs/0.12/patterns/celery/.

def make_celery(app):
    celery = Celery(app.import_name, broker=app.config['CELERY_BROKER_URL'])
    celery.conf.update(app.config)
    task_base = celery.Task

    class ContextTask(task_base):
        abstract = True

        def __call__(self, *args, **kwargs):
            if current_app.config['TESTING']:
                with app.test_request_context():
                    return task_base.__call__(self, *args, **kwargs)
            with app.app_context():
                return task_base.__call__(self, *args, **kwargs)

    celery.Task = ContextTask
    return celery

celery = make_celery(current_app)


The __call__ method ensures that celery task is provided with proper app context to work with.

 

Event-driven programming in Flask with Blinker signals

Setting up blinker:

The Open Event Project offers event managers a platform to organize all kinds of events including concerts, conferences, summits and regular meetups. In the server part of the project, the issue at hand was to perform multiple tasks in background (we use celery for this) whenever some changes occurred within the event, or the speakers/sessions associated with the event.

The usual approach to this would be applying a function call after any relevant changes are made. But the statements making these changes were distributed all over the project at multiple places. It would be cumbersome to add 3-4 function calls (which are irrelevant to the function they are being executed) in so may places. Moreover, the code would get unstructured with this and it would be really hard to maintain this code over time.

That’s when signals came to our rescue. From Flask 0.6, there is integrated support for signalling in Flask, refer http://flask.pocoo.org/docs/latest/signals/ . The Blinker library is used here to implement signals. If you’re coming from some other language, signals are analogous to events.

Given below is the code to create named signals in a custom namespace:


from blinker import Namespace

event_signals = Namespace()
speakers_modified = event_signals.signal('event_json_modified')

If you want to emit a signal, you can do so by calling the send() method:


speakers_modified.send(current_app._get_current_object(), event_id=event.id, speaker_id=speaker.id)

From the user guide itself:

“ Try to always pick a good sender. If you have a class that is emitting a signal, pass self as sender. If you are emitting a signal from a random function, you can pass current_app._get_current_object() as sender. “

To subscribe to a signal, blinker provides neat decorator based signal subscriptions.


@speakers_modified.connect
def name_of_signal_handler(app, **kwargs):

 

Some Design Decisions:

When sending the signal, the signal may be sending lots of information, which your signal may or may not want. e.g when you have multiple subscribers listening to the same signal. Some of the information sent by the signal may not be of use to your specific function. Thus we decided to enforce the pattern below to ensure flexibility throughout the project.


@speakers_modified.connect
def new_handler(app, **kwargs):
# do whatever you want to do with kwargs['event_id']

In this case, the function new_handler needs to perform some task solely based on the event_id. If the function was of the form def new_handler(app, event_id), an error would be raised by the app. A big plus of this approach, if you want to send some more info with the signal, for the sake of example, if you also want to send speaker_name along with the signal, this pattern ensures that no error is raised by any of the subscribers defined before this change was made.

When to use signals and when not ?

The call to send a signal will of course be lying in another function itself. The signal and the function should be independent of each other. If the task done by any of the signal subscribers, even remotely affects your current function, a signal shouldn’t be used, use a function call instead.

How to turn off signals while testing?

When in testing mode, signals may slow down your testing as unnecessary signals subscribers which are completely independent from the function being tested will be executed numerous times. To turn off executing the signal subscribers, you have to make a small change in the send function of the blinker library.

Below is what we have done. The approach to turn it off may differ from project to project as the method of testing differs. Refer https://github.com/jek/blinker/blob/master/blinker/base.py#L241 for the original function.


def new_send(self, *sender, **kwargs):
    if len(sender) == 0:
        sender = None
    elif len(sender) > 1:
        raise TypeError('send() accepts only one positional argument, '
                        '%s given' % len(sender))
    else:
        sender = sender[0]
    # only this line was changed
    if not self.receivers or app.config['TESTING']:
        return []
    else:
        return [(receiver, receiver(sender, **kwargs))
                for receiver in self.receivers_for(sender)]
                
Signal.send = new_send

event_signals = Namespace
# and so on ....

That’s all for now. Have some fun signaling 😉 .

 

Ticket Ordering and Positioning (Front-end)

As discussed in my last blog about ticket ordering and positioning, in this blog we are gonna talk about how we implement the front-end part of re-arranging the tickets. We essentially do it using compute and methods of Vue.js. The functionality that is expected in the front-end is, the event organizer should be able to move the tickets Up or Down the order and save that position so that it gets displayed later in that very particular order.

Like I said above we use two main things of Vue.JS for this purpose – Compute and Methods.

Compute

We use this to get the sorted list of tickets based on the position key of the tickets and use this sorted list to display the tickets in the event editing wizard. Whenever you change the value of the position for a ticket, it automatically updates the list to sorted list again and hence the order of ticket in the front-end also updates. To add a compute function in Vue.JS, inside the new Vue() object creation, we add an attribute computed and inside that we put all the functions that we are gonna use. So in our case the function is sortedTickets . We make use of the sort function of lodash to sort the tickets array based on it’s position attribute.

Now while showing or looping over the tickets, we loop over sortedTickets  rather than the original ticket array.

Method

This method is called when the button is clicked to move it up or down. This makes the calculations to determine the values of the position of the two tickets which are being re-ordered in a single click. To add a method, we do it in a similar way like computed but using methods attribute instead. The methods we have written to move tickets up or down is moveTicket.

It has 3 parameters – ticket, index and direction. So when this function call is emitted, depending on the button, the direction is either “up” or “down” while the other two parameters are the ticket properties of the particular ticket. So in this function we check the direction and accordingly update the value of position for the tickets involved in the arranging. As we update the position here, because of the compute, the UI automatically updates to show the tickets in the new order.

Finally after all arrangement is done, the position is always saved in a hidden input field which is then passed as form data and is saved in the database so that the position value can be used in other pages for showing the ticket in order.

Show Ordered Ticket

In other pages, while showing ordered ticket, we already receive the ticket array in sorted format based on the position key. Hence, we don’t need to worry about it in the front-end and it automatically is shown in the sorted order.