Handling date-time in the Open Event Project

Handling date time in code becomes little tricky when the project is used internationally because then there comes additional term Timezone. Timezone is a property of a location which needs to be considered while comparing that time with the time in another location. For example – there are two villages A and B. One day Ram from village A calls his friend Shyam in village B at 8:00 am to wish “good morning”. But Shyam receives Ram’s call at 6pm on same day and he replies “good evening”. That means village A’s timezone is 10 hrs  behind village B’s timezone. So here we need some reference timezone about which all other timezones can be declared. This is where UTC (Coordinated Universal Time) comes into play. UTC is reference timezone by which all the timezones are declared. For example – Indian timezone is 5 hrs and 30 mins ahead of UTC which is denoted as UTC+05:30. In languages, these timezones are declared in date time library using constants such as ‘Asia/Kolkata’ which is Indian Standard Time. I will be talking about working with  date time in python in this blog. In the FOSSASIA’s Open Event project since it is event management system, handling date-time with the timezone is one of the important tasks.

Here is the relevant code:

>>> import datetime
>>> import pytz
>>> now = datetime.datetime.now()

datetime.datetime.now()  returns naive datetime as of the time setting of the machine on which the code is running. Naive date means it doesn’t contain any info about the timezone. It just contains some number values of year, month, hours etc. So just by looking at naive date we cannot actually understand the time. There comes aware datetime which contains timezone info.

>> now
datetime.datetime(2017, 5, 12, 21, 46, 16, 909983)
>>> now.isoformat()
'2017-05-12T21:46:16.909983'
>>> aware_now = pytz.timezone('Asia/Kolkata').localize(now)
>>> aware_now
datetime.datetime(2017, 5, 12, 21, 46, 16, 909983, tzinfo=<DstTzInfo 'Asia/Kolkata' IST+5:30:00 STD>

Pytz provides timezone object which takes string argument for timezone which has localize method which adds timezone info to the datetime object. Hence now aware datetime has timezone info too. Now if we print the time.

>>> aware_now.isoformat()
'2017-05-12T21:46:16.909983+05:30

We get the +05:30 extra string at the end which gives timezone info. +05:30 means the timezone is 5 hrs and 30 mins ahead of UTC timezone. The comparison between datetimes can be made between naive-naive and aware-aware. If we try to compare between naive and aware,

>>> now < aware_now
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: can't compare offset-naive and offset-aware datetimes

>>> now2 = datetime.datetime.now()
>>> now2
datetime.datetime(2017, 5, 15, 9, 44, 25, 990666)
>>> now < now2
True
>>> aware_now.tzinfo
<DstTzInfo 'Asia/Kolkata' IST+5:30:00 STD>

tzinfo carries timezone info of the datetime object. We can make aware date to unaware by method replacing tzinfo to None.

>>> unaware_now = aware_now.replace(tzinfo=None)
>>> unaware_now.isoformat()
'2017-05-12T21:46:16.909983'

Formating datetime is done by mostly these two methods. One of which takes string format in which the result is required and another returns datetime in iso format.

>>> now.strftime('%Y-%m-%dT%H:%M:%S%z')
'2017-05-12T21:46:16'
>>> aware_now.strftime('%Y-%m-%dT%H:%M:%S%z')
'2017-05-12T21:46:16+0530'

>>> now.time().isoformat()
'21:46:16.909983'
>>> now.date().isoformat()
'2017-05-12'

>>> now_in_brazil_east = datetime.datetime.now(pytz.timezone('Brazil/East'))
>>> now_in_brazil_east.isoformat()
'2017-05-15T06:49:51.311012-03:00'

We can pass timezone argument to the now method to get current time in the passed timezone. But the care must be taken as this will use the timezone setting of the machine on which code is running to calculate the time at the supplied timezone.

Application

In the FOSSASIA’s Open Event Project, date-time is taken from the user along with the timezone like in one of the example shown below.

Date Time Getter Open Event
Date Time Getter in Create Event Step 1 Open Event Front End

This is part of the event creation page where user has to provide date-time along with the timezone choice. At back-end the date-time and timezone are stored separately in the database. The event model looks like

class Event(db.Model):
 """Event object table"""
 ...
 start_time = db.Column(db.DateTime, nullable=False)
 end_time = db.Column(db.DateTime, nullable=False)
 timezone = db.Column(db.String, nullable=False, default="UTC")
 ...

The comparison between the stored date-time info with the real time cannot be done directly. Since the timezones of the both times need to be considered along with the date-time values while comparison. Likewise there is one case in the project code where tickets are filtered based on the start time.

def get_sales_open_tickets(event_id, event_timezone='UTC'):
  tickets = Ticket.query.filter(Ticket.event_id == event_id).filter(
      Ticket.sales_start <= datetime.datetime.now(pytz.timezone(event_timezone)).replace(tzinfo=None)).filter(
      Ticket.sales_end >= datetime.datetime.now(pytz.timezone(event_timezone)).replace(tzinfo=None))
  …

In this case, first current time is found out using timezone method in the timezone which is stored as a separate data field in the database. Since comparison cannot be done between aware and naive date-time. Hence once current date-time is found out in the user’s timezone, it is made naive using replace method which makes the aware date-time into naive again. Hence can be compared with the naive date-time stored already.

Continue Reading

Sorting language-translation in Open Event Server project using Jinja 2 dictsort.

Working on the Open Event Server project an issue about arranging language-translation listing in alphabetical order came up. To solve this issue of language listing arrangement i.e. #2817, I found the ‘d0_dictsort’ function in jinja2 to sort dictionaries. It is a defined in jinja2.filters. Python dicts are unsorted and in our web application we at times may want to order them by either their key or value. So this function comes handy.

This is what the function looks like:

do_dictsort(value, case_sensitive=False, by='key')

We can write them in three ways as:

{% for record in my_dictionary|dictsort %}
    case insensitive and sort the dict by key

{% for record in my_dictionary|dicsort(true) %}
    case sensitive and sort the dict by key

{% for record in my_dictionary|dictsort(false, 'value') %}
    sort the dict by value, normally sorted and case insensitive
  1.       The first way is easily understood that dict has been sorted by key not taking case into consideration. It is just in the same way written as dictsort(false).
  2.       Second way is basically the first being case sensitive. dictsort(true) here tells us that case is sensitive.
  3.      Third way is dictsort(false,’value’). The first parameter defines that case insensitive while second parameter defines that it is sorted by ‘value’.

The issues was to sort translation selector for the page in alphabetical order. The languages were stored in a dictionary which to change in order, I found this function very easy and useful.

Basically what we had was:

This is how the function was used in the code for the sort. Like this:

<ul class="dropdown-menu lang-list">
   {% for code in all-languages|dictsort(false,'value') %}
       <li><a  href="#" style="#969191" class="translate" id="{{ code[0] }}">{{  all_languages[code[0]] }}<>a><li>
    {% endfor %}
<ul>


Here:
{{ all_languages }} is the list which contained the languages like French, English, etc., which could be accessed with its global language code. code here(index for all_languages) is a tuple of {‘global_language_code’,’language’} (An example would be (‘fr’,’French’), so code[0] gave me the language_code.

Finally, the result:

This is one of the simple ways to sort your dictionaries.

Continue Reading

Open Event Server: No (no-wrap) Ellipsis using jquery!

Yes, the title says it all i.e., Enabling multiple line ellipsis. This was used to solve an issue to keep Session abstract view within 200 characters (#3059) on FOSSASIA‘s Open Event Server project.

There is this one way to ellipsis a paragraph in html-css and that is by using the text-overflow property:

.div_class{
white-space: nowrap;
overflow: hidden;
text-overflow: ellipsis;
}’’

But the downside of this is the one line ellipis. Eg: My name is Medozonuo. I am…..

And here you might pretty much want to ellipsis after a few characters in multiple lines, given that your div space is small and you do want to wrap your paragraph. Or maybe not.

So jquery to the rescue.

There are two ways you can easily do this multiple line ellipsis:

1) Height-Ellipsis (Using the do-while loop):

//script:
if ($('.div_class').height() > 100) {
    var words = $('.div_class').html().split(/\s+/);
    words.push('...');

    do {
        words.splice(-2, 1);
        $('.div_class').html( words.join(' ') );
    } while($('.div_class').height() > 100);
}

Here, you check for the div content’s height and split the paragraph after that certain height and add a “…”, do- while making sure that the paragraphs are in multiple lines and not in one single line. But checkout for that infinite loop.

2) Length-Ellipsis (Using substring function):  

//script:
$.each($('.div_class'), function() {
        if ($(this).html().length > 100) {
               var cropped_words = $(this).html();
               cropped_words = cropped_words.substring(0, 200) + "...";
               $(this).html(cropped_words);
        }
 });

Here, you check for the length/characters rather than the height, take in the substring of the content starting from 0-th character to the 200-th character and then add in extra “…”.

This is exactly how I used it in the code.

$.each($('.short_abstract',function() {
   if ($(this).html().length > 200) {
       var  words = $(this).html();
       words = words.substring(0,200 + "...";
       $(this).html(words);
    }
});


So ellipsing paragraphs over heights and lengths can be done using jQuery likewise.

Continue Reading

DetachedInstanceError: Dealing with Celery, Flask’s app context and SQLAlchemy in the Open Event Server

In the open event server project, we had chosen to go with celery for async background tasks. From the official website,

What is celery?

Celery is an asynchronous task queue/job queue based on distributed message passing.

What are tasks?

The execution units, called tasks, are executed concurrently on a single or more worker servers using multiprocessing.

After the tasks had been set up, an error constantly came up whenever a task was called

The error was:

DetachedInstanceError: Instance <User at 0x7f358a4e9550> is not bound to a Session; attribute refresh operation cannot proceed

The above error usually occurs when you try to access the session object after it has been closed. It may have been closed by an explicit session.close() call or after committing the session with session.commit().

The celery tasks in question were performing some database operations. So the first thought was that maybe these operations might be causing the error. To test this theory, the celery task was changed to :

@celery.task(name='lorem.ipsum')
def lorem_ipsum():
    pass

But sadly, the error still remained. This proves that the celery task was just fine and the session was being closed whenever the celery task was called. The method in which the celery task was being called was of the following form:

def restore_session(session_id):
    session = DataGetter.get_session(session_id)
    session.deleted_at = None
    lorem_ipsum.delay()
    save_to_db(session, "Session restored from Trash")
    update_version(session.event_id, False, 'sessions_ver')


In our app, the app_context was not being passed whenever a celery task was initiated. Thus, the celery task, whenever called, closed the previous app_context eventually closing the session along with it. The solution to this error would be to follow the pattern as suggested on http://flask.pocoo.org/docs/0.12/patterns/celery/.

def make_celery(app):
    celery = Celery(app.import_name, broker=app.config['CELERY_BROKER_URL'])
    celery.conf.update(app.config)
    task_base = celery.Task

    class ContextTask(task_base):
        abstract = True

        def __call__(self, *args, **kwargs):
            if current_app.config['TESTING']:
                with app.test_request_context():
                    return task_base.__call__(self, *args, **kwargs)
            with app.app_context():
                return task_base.__call__(self, *args, **kwargs)

    celery.Task = ContextTask
    return celery

celery = make_celery(current_app)


The __call__ method ensures that celery task is provided with proper app context to work with.

 

Continue Reading

Event-driven programming in Flask with Blinker signals

Setting up blinker:

The Open Event Project offers event managers a platform to organize all kinds of events including concerts, conferences, summits and regular meetups. In the server part of the project, the issue at hand was to perform multiple tasks in background (we use celery for this) whenever some changes occurred within the event, or the speakers/sessions associated with the event.

The usual approach to this would be applying a function call after any relevant changes are made. But the statements making these changes were distributed all over the project at multiple places. It would be cumbersome to add 3-4 function calls (which are irrelevant to the function they are being executed) in so may places. Moreover, the code would get unstructured with this and it would be really hard to maintain this code over time.

That’s when signals came to our rescue. From Flask 0.6, there is integrated support for signalling in Flask, refer http://flask.pocoo.org/docs/latest/signals/ . The Blinker library is used here to implement signals. If you’re coming from some other language, signals are analogous to events.

Given below is the code to create named signals in a custom namespace:


from blinker import Namespace

event_signals = Namespace()
speakers_modified = event_signals.signal('event_json_modified')

If you want to emit a signal, you can do so by calling the send() method:


speakers_modified.send(current_app._get_current_object(), event_id=event.id, speaker_id=speaker.id)

From the user guide itself:

“ Try to always pick a good sender. If you have a class that is emitting a signal, pass self as sender. If you are emitting a signal from a random function, you can pass current_app._get_current_object() as sender. “

To subscribe to a signal, blinker provides neat decorator based signal subscriptions.


@speakers_modified.connect
def name_of_signal_handler(app, **kwargs):

 

Some Design Decisions:

When sending the signal, the signal may be sending lots of information, which your signal may or may not want. e.g when you have multiple subscribers listening to the same signal. Some of the information sent by the signal may not be of use to your specific function. Thus we decided to enforce the pattern below to ensure flexibility throughout the project.


@speakers_modified.connect
def new_handler(app, **kwargs):
# do whatever you want to do with kwargs['event_id']

In this case, the function new_handler needs to perform some task solely based on the event_id. If the function was of the form def new_handler(app, event_id), an error would be raised by the app. A big plus of this approach, if you want to send some more info with the signal, for the sake of example, if you also want to send speaker_name along with the signal, this pattern ensures that no error is raised by any of the subscribers defined before this change was made.

When to use signals and when not ?

The call to send a signal will of course be lying in another function itself. The signal and the function should be independent of each other. If the task done by any of the signal subscribers, even remotely affects your current function, a signal shouldn’t be used, use a function call instead.

How to turn off signals while testing?

When in testing mode, signals may slow down your testing as unnecessary signals subscribers which are completely independent from the function being tested will be executed numerous times. To turn off executing the signal subscribers, you have to make a small change in the send function of the blinker library.

Below is what we have done. The approach to turn it off may differ from project to project as the method of testing differs. Refer https://github.com/jek/blinker/blob/master/blinker/base.py#L241 for the original function.


def new_send(self, *sender, **kwargs):
    if len(sender) == 0:
        sender = None
    elif len(sender) > 1:
        raise TypeError('send() accepts only one positional argument, '
                        '%s given' % len(sender))
    else:
        sender = sender[0]
    # only this line was changed
    if not self.receivers or app.config['TESTING']:
        return []
    else:
        return [(receiver, receiver(sender, **kwargs))
                for receiver in self.receivers_for(sender)]
                
Signal.send = new_send

event_signals = Namespace
# and so on ....

That’s all for now. Have some fun signaling 😉 .

 

Continue Reading

Ticket Ordering and Positioning (Front-end)

As discussed in my last blog about ticket ordering and positioning, in this blog we are gonna talk about how we implement the front-end part of re-arranging the tickets. We essentially do it using compute and methods of Vue.js. The functionality that is expected in the front-end is, the event organizer should be able to move the tickets Up or Down the order and save that position so that it gets displayed later in that very particular order.

Like I said above we use two main things of Vue.JS for this purpose – Compute and Methods.

Compute

We use this to get the sorted list of tickets based on the position key of the tickets and use this sorted list to display the tickets in the event editing wizard. Whenever you change the value of the position for a ticket, it automatically updates the list to sorted list again and hence the order of ticket in the front-end also updates. To add a compute function in Vue.JS, inside the new Vue() object creation, we add an attribute computed and inside that we put all the functions that we are gonna use. So in our case the function is sortedTickets . We make use of the sort function of lodash to sort the tickets array based on it’s position attribute.

Now while showing or looping over the tickets, we loop over sortedTickets  rather than the original ticket array.

Method

This method is called when the button is clicked to move it up or down. This makes the calculations to determine the values of the position of the two tickets which are being re-ordered in a single click. To add a method, we do it in a similar way like computed but using methods attribute instead. The methods we have written to move tickets up or down is moveTicket.

It has 3 parameters – ticket, index and direction. So when this function call is emitted, depending on the button, the direction is either “up” or “down” while the other two parameters are the ticket properties of the particular ticket. So in this function we check the direction and accordingly update the value of position for the tickets involved in the arranging. As we update the position here, because of the compute, the UI automatically updates to show the tickets in the new order.

Finally after all arrangement is done, the position is always saved in a hidden input field which is then passed as form data and is saved in the database so that the position value can be used in other pages for showing the ticket in order.

Show Ordered Ticket

In other pages, while showing ordered ticket, we already receive the ticket array in sorted format based on the position key. Hence, we don’t need to worry about it in the front-end and it automatically is shown in the sorted order.

Continue Reading

Using Cloud storage for event exports

Open-event orga server provides the ability to the organizer to create a complete export of the event they created. Currently, when an organizer triggers the export in orga server, A celery job is set to complete the export task resulting asynchronous completion of the job. Organizer gets the download button enabled once export is ready.

Till now the main issue was related to storage of those export zip files. All exported zip files were stored directly in local storage and that even not by using storage module created under orga server.

local storage path

On a mission to solve this, I made three simple steps that I followed to solve this issue.

These three steps were:

  1. Wait for shutil.make_archive to complete archive and store it in local storage.
  2. Copy the created archive to storage ( specified by user )
  3. Delete local archive created.

The easiest part here was to make these files upload to different storage ( s3, gs, local) as we already have storage helper

def upload(uploaded_file, key, **kwargs):
    """
    Upload handler
    """

The most important logic of this issue resides to this code snippet.

    dir_path = dir_path + ".zip"
 
     storage_path = UPLOAD_PATHS['exports']['zip'].format(
         event_id = event_id
     )
     uploaded_file = UploadedFile(dir_path, dir_path.rsplit('/', 1)[1])
     storage_url = upload(uploaded_file, storage_path)
 
    if get_settings()['storage_place'] != "s3" or get_settings()['storage_place'] != 'gs':
        storage_url = app.config['BASE_DIR'] + storage_url.replace("/serve_","/")
    return storage_url

From above snippet, it is clear that we are extending the process of creating the zip. Once the zip is created we will make storage path for cloud storage and upload it. Only one thing will take the time to understand here is the last second and third line of above snippet.

if get_settings()['storage_place'] != "s3" or get_settings()['storage_place'] != 'gs':
        storage_url = app.config['BASE_DIR'] + storage_url.replace("/serve_","/")

Initial the plan was simple to serve the files through “serve_static” but then the test cases were expecting a file at this location thus I had to remove “serve_” part for local storage and then it works fine on those three steps.

Next thing on this storage process need to be discussed is the feature to delete old exports. I believe one reason to keep them would be an old backup of your event will be always there with us at our cloud storage.

Continue Reading

Ticket Ordering or Positioning (back-end)

One of the many feature requests that we got for our open event organizer server or the eventyay website is ticket ordering. The event organizers wanted to show the tickets in a particular order in the website and wanted to control the ordering of the ticket. This was a common request by many and also an important enhancement. There were two main things to deal with when ticket ordering was concerned. Firstly, how do we store the position of the ticket in the set of tickets. Secondly, we needed to give an UI in the event creation/edit wizard to control the order or position of a ticket. In this blog, I will talk about how we store the position of the tickets in the backend and use it to show in our public page of the event.

Continue Reading

PIL to convert type and quality of image

Image upload is an important part of the server. The images can be in different formats and after applying certain javascript modifications, they can be changed to different formats. For example, when an image is uploaded after cropping in open event organizer server, it is saved in PNG format. But PNG is more than 5 times larger than JPEG image. So when we upload a 150KB image, the image finally reaching the server is around 1MB which is huge. So we need to decide in the server which image format to select in different cases and how to convert them.

Continue Reading

Adding Google Analytics To All Pages Using Flask

Google Analytics gives a detailed insight about your website including how many people visited, time, demography, how many returning visitors and all such information. It’s a real important tool to have. All you have to do is create a Universal Analytics Tracking code and use it in a javascript code. The only problem is this code needs to be present in all the pages that you wants the analytics data for. So changing any part of the javascript code anytime, needs to be changed in all .html files.

However, there is a better way of doing it in flask. Create a file base.html and write the code:

<script>
(function(i,s,o,g,r,a,m){i[‘GoogleAnalyticsObject’]=r;i[r]=i[r]||function(){
(i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
})(window,document,’script’,’https://www.google-analytics.com/analytics.js’,’ga’);

ga(‘create’, ‘<track-code>’, ‘auto’);
ga(‘send’, ‘pageview’);

</script>

Then using the property of jinja2 template extend this file in all the html files, i.e. {% extends ‘gentelella/admin/base.html’ %}. Thus now when you make some change in the above mentioned javascrpt code, you need to change it only in one place and it is changed in all other places.

Continue Reading
Close Menu