Generating xCal calendar in python

{ Repost from my personal blog @ https://blog.codezero.xyz/generate-xcal-calendar-in-python }

“xCal”, is an XML format for iCalendar data.

The iCalendar data format (RFC5545) is a widely deployed interchange format for calendaring and scheduling data.

A Sample xCal document

<?xml version="1.0" encoding="utf-8"?>  
<iCalendar xmlns:xCal="urn:ietf:params:xml:ns:xcal">  
    <vcalendar>
        <version>2.0</version>
        <prodid>-//Pentabarf//Schedule 1.0//EN</prodid>
        <x-wr-caldesc>FOSDEM 2016</x-wr-caldesc>
        <x-wr-calname>Schedule for events at FOSDEM 2016</x-wr-calname>
        <vevent>
            <method>PUBLISH</method>
            <uid>123e4567-e89b-12d3-a456-426655440000</uid>
            <dtstart>20160131T090000</dtstart>
            <dtend>20160131T091000</dtend>
            <duration>00:10:00:00</duration>
            <summary>Introduction to the SDR Track- Speakers, Topics, Algorithm</summary>
            <description>&lt;p&gt;The opening talk for the SDR devroom at FOSDEM 2016.&lt;/p&gt;</description>
            <class>PUBLIC</class>
            <status>CONFIRMED</status>
            <categories>Software Defined Radio</categories>
            <url>https:/fosdem.org/2016/schedule/event/sdrintro/</url>
            <location>AW1.125</location>
            <attendee>Martin Braun</attendee>
        </vevent>
    </vcalendar>
</iCalendar>

Each event/session will be in a seperate vevent block. Each speaker/attendee of an event/session will be in an attendee block inside a vevent block.

Some important elements are:

  1. version – Has the version of the iCalendar data
  2. prodid – Contains the name of the application/generator that generated this document
  3. x-wr-caldesc – A descriptive name for this calendar
  4. x-wr-calname – A description of the calendar

The structure and keywords used in xCal are the same as those used in the iCal format. To generate the XML document, we’ll be using python’s ElementTreeXML API that is part of the Python standard library.

We’ll be using two main classes of the ElementTree API:

  1. Element – used to create a standard node. (Used for the root node)
  2. SubElement – used to create a sub element and attache the new node to a parent

Let’s start with the root iCalendar node and set the required attributes.

from xml.etree.ElementTree import Element, SubElement, tostring

i_calendar_node = Element('iCalendar')  
i_calendar_node.set('xmlns:xCal', 'urn:ietf:params:xml:ns:xcal')

Now, to add the vcalendar node to the iCalendar node.

v_calendar_node = SubElement(i_calendar_node, 'vcalendar')

Let’s add the other aspects of the calendar to the vcalendar node as separate sub nodes.

version_node = SubElement(v_calendar_node, 'version')  
version_node.text = '2.0'

prod_id_node = SubElement(v_calendar_node, 'prodid')  
prod_id_node.text = '-//fossasia//open-event//EN'

cal_desc_node = SubElement(v_calendar_node, 'x-wr-caldesc')  
cal_desc_node.text = "Calendar"

cal_name_node = SubElement(v_calendar_node, 'x-wr-calname')  
cal_name_node.text = "Schedule for sessions"

Now, we have added information about our calendar. Now to add the actual events to the calendar. Each event would be a vevent node, a child of vcalendar node. We can loop through all our available event/sessions and add them to the calendar.

for session in sessions:  
    v_event_node = SubElement(v_calendar_node, 'vevent')

    uid_node = SubElement(v_event_node, 'uid')
    uid_node.text = str(session.id)

    dtstart_node = SubElement(v_event_node, 'dtstart')
    dtstart_node.text = session.start_time.isoformat()

    dtend_node = SubElement(v_event_node, 'dtend')
    dtend_node.text = tz.localize(session.end_time).isoformat()

    duration_node = SubElement(v_event_node, 'duration')
    duration_node.text =  "00:30"

    summary_node = SubElement(v_event_node, 'summary')
    summary_node.text = session.title

    description_node = SubElement(v_event_node, 'description')
    description_node.text = session.short_abstract

    class_node = SubElement(v_event_node, 'class')
    class_node.text = 'PUBLIC'

    status_node = SubElement(v_event_node, 'status')
    status_node.text = 'CONFIRMED'

    categories_node = SubElement(v_event_node, 'categories')
    categories_node.text = session.session_type.name

    url_node = SubElement(v_event_node, 'url')
    url_node.text = "https://some.conf/event/" + str(session.id)

    location_node = SubElement(v_event_node, 'location')
    location_node.text = session.microlocation.name

    for speaker in session.speakers:
        attendee_node = SubElement(v_event_node, 'attendee')
        attendee_node.text = speaker.name

Please note that all the timings in the XML Document must comply with ISO 8601 and must have the date+time+timezone. Example: 2007-04-05T12:30-02:00.

We’re still not done yet. We now have the XML document as an Element object. But we’ll be needing it as a string to either store it somewhere or display it.

The document can be converted to a string by using the ElementTree API’s tostring helper method and passing the root node.

xml_as_string = tostring(i_calendar_node)

And that’s it. You now have a proper XML document representing your events.

PIL to convert type and quality of image

Image upload is an important part of the server. The images can be in different formats and after applying certain javascript modifications, they can be changed to different formats. For example, when an image is uploaded after cropping in open event organizer server, it is saved in PNG format. But PNG is more than 5 times larger than JPEG image. So when we upload a 150KB image, the image finally reaching the server is around 1MB which is huge. So we need to decide in the server which image format to select in different cases and how to convert them.

Continue reading PIL to convert type and quality of image

File upload progress in a Node app using Socket.io

If you look at the webapp generator, you’ll see that there is an option to upload a zip file containing event data. We wanted to give visual cue to the user when he is uploading to see how much file has uploaded.

We are uploading the file, and giving the generate start command via socket.io events instead of POST requests here.

To observe file upload progress on socket (when sending file using a Buffer), there is an awesome node module available called socketio-upload-progress.

In our webapp you can see we implemented it on the frontend here in the form.js and here in the backend in app.js

Basically on the backend you should add the socketio-file-upload module as a middleware to express

var siofu = require("socketio-file-upload");
var app = express()
    .use(siofu.router)
    .listen(8000);

After a socket is opened, set up the upload directory and start listening for uploads

io.on("connection", function(socket){
    var uploader = new siofu();
    uploader.dir = "/path/to/save/uploads";
    uploader.listen(socket);
});

On the frontend, we’ll listen for an input change on an file input type element whose id is siofu_upload

var socket = io.connect();
var uploader = new SocketIOFileUpload(socket);
uploader.listenOnInput(document.getElementById("siofu_input"));

One thing to note here is that, if you observe percentage of upload on frontend, it’ll give you false values. The correct values of how much data is actually transferred can be found in the backend. So observe progress in backend, and send percentage to frontend using the same socket.

  uploader.on('progress', function(event) {
    console.log(event.file.bytesLoaded / event.file.size)
    socket.emit('upload.progress', {
      percentage:(event.file.bytesLoaded / event.file.size) * 100
    })
});

 

Adding Google Analytics To All Pages Using Flask

Google Analytics gives a detailed insight about your website including how many people visited, time, demography, how many returning visitors and all such information. It’s a real important tool to have. All you have to do is create a Universal Analytics Tracking code and use it in a javascript code. The only problem is this code needs to be present in all the pages that you wants the analytics data for. So changing any part of the javascript code anytime, needs to be changed in all .html files.

However, there is a better way of doing it in flask. Create a file base.html and write the code:

<script>
(function(i,s,o,g,r,a,m){i[‘GoogleAnalyticsObject’]=r;i[r]=i[r]||function(){
(i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
})(window,document,’script’,’https://www.google-analytics.com/analytics.js’,’ga’);

ga(‘create’, ‘<track-code>’, ‘auto’);
ga(‘send’, ‘pageview’);

</script>

Then using the property of jinja2 template extend this file in all the html files, i.e. {% extends ‘gentelella/admin/base.html’ %}. Thus now when you make some change in the above mentioned javascrpt code, you need to change it only in one place and it is changed in all other places.

Getting started with Docker Compose

In this post, I will talk about running multiple containers at once using Docker Compose.

The problem ?

Suppose you have a complex app with Database containers, Redis and what not. How are you going to start the app ? One way is to write a shell script that starts the containers one by one.

docker run postgres:latest --name mydb -d
docker run redis:3-alpine --name myredis -d
docker run myapp -d

Now suppose these containers have lots of configurations (links, volumes, ports, environment variables) that they need to function. You will have to write those parameters in the shell script.

docker network create myapp_default
docker run postgres:latest --name db -d -p 5432:5432 --net myapp_default
docker run redis:3-alpine --name redis -d -p 6379:6379 \
    --net myapp_default -v redis:/var/lib/redis/data
docker run myapp -d -p 5000:5000 --net myapp_default -e SOMEVAR=value \
    --link db:db --link redis:redis -v storage:/myapp/static

Won’t it get un-manageable ? Won’t it be great if we had a cleaner way to running multiple containers. Here comes docker-compose to the rescue.

Docker compose

Docker compose is a python package which does the job of handling multiple containers for an application very elegantly. The main file of docker-compose is docker-compose.yml which is a YAML like syntax file with the settings/components required to run your app. Once you define that file, you can just do docker-compose up to start your app with all the components and settings. Pretty cool, right ?

So let’s see the docker-compose.yml for the fictional app we have considered above.

version: '2'

services:
  db:
    image: postgres:latest
    ports:
      - '5432:5432'

  redis:
    image: 'redis:3-alpine'
    command: redis-server
    volumes:
      - 'redis:/var/lib/redis/data'
    ports:
      - '6379:6379'

  web:
    build: .
    environment:
      SOMEVAR: value
    links:
      - db:db
      - redis:redis
    volumes:
      - 'storage:/myapp/static'
    ports:
      - '5000:5000'

volumes:
  redis:
  storage:

Once this file is in the project’s root directory, you can use docker-compose up to start the application. It will run the services in the order in which they have been defined in the YAML file.

Docker compose has a lot of commands that generally correspond to the parameters that docker runaccepts. You can see a full list on the official docker-compose reference.

Conclusion

It’s no doubt that docker-compose is a boon when you have to run complex applications. It personally use Compose in every dockerized application that I write. In GSoC 16, I dockerized Open Event. Here is the docker-compose.yml file if you are interested.

PS – If you liked this post, you might find my other posts on Docker interesting. Do take a look and let me know your views.

 

{{ Repost from my personal blog http://aviaryan.in/blog/gsoc/docker-compose-starting.html }}

KISS Datatable

Recenlty I’ve faced a problem with sorting columns in Datatable.

What is Datatable?

Datatable is a plug-in for Jquery library. It provides a lot of features like pagination, quick search or multi-column ordering. Besides, you can develop easily Bootstrap or Foundation ui css styles. There are also more other option but It doesn’t make sense to list it here, because you can visit their site and you can read clearly documentation. On Datatable website you can see a lot of examples. First of them shows how to improve your ordinary table to awesome and rich of features table. One function changes everything, It’s fantastic!  

$('#myTable').DataTable();

Returning to my problem which I’ve faced, as I told it was problem related to sorting column in table.

I know sorting is a trivial thing. I hope that everyone knows it 🙂 Sorting by a date is also implemented in a datatable library. So everything is clear when we don’t change date format to human readable format. I mean something like this ‘3 hours ago’, ‘1 year ago’.

When Open Event team tested how datatable manages ordering columns in that format it didn’t work. It’s quite hard to sort by that format, So I’ve invented an idea. Surely you are wondering what i’ve invented. I’ve postponed my minds about sort by this values. It can direct to overwork. When I thought about it, weird ideas came to my mind, a lots of conditions, If statements… Therefore I’ve resigned from this. I’ve used KISS principle. KISS means ‘keep it simple stupid’. I like it!

Therefore that sorting is implemented on frontend side. I’ve decided not to display human readable date format at the beginning. I find  all dates which have format “YYYY-MM-DD HH:mm” then I replace that format to human readable format. So it’s very quick and comfortable, and doesn’t require a lot conditions to write. Of course I’ve tried to implement it in Datatable library. I suppose that it would  require more effort than it’s now.

Below You can see great function which changes a date in frontend side but does not change data in a datatable. So sorting process takes place in a datatable using format  “YYYY-MM-DD HH:mm” but user see human readable format. Isn’t it awesome?!

function change_column_time_to_humanize_format(datatable_name, column_id) {
  $(datatable_name).each(function( key, value ) {
    $(value).children().each(function( key1, value2 ) {
       if(key1 === column_id ){
          var current_value = $(value2).text().slice(0, 16);
          var changed_value = moment(current_value, "YYYY-MM-DD hh:mm").fromNow()
          var isValid = moment(current_value, "YYYY-MM-DD HH:mm", true).isValid()
          if (changed_value !== current_value && isValid === true){
              $(value2).text(changed_value)
          }
      }
  });
});

 

Building the Scheduler UI

{ Repost from my personal blog @ https://blog.codezero.xyz/building-the-scheduler-ui }

If you hadn’t already noticed, Open Event has got a shiny new feature. A graphical and an Interactive scheduler to organize sessions into their respective rooms and timings.

As you can see in the above screenshot, we have a timeline on the left. And a lot of session boxes to it’s right. All the boxes are re-sizable and drag-drop-able. The columns represent the different rooms (a.k.a micro-locations). The sessions can be dropped into their respective rooms. Above the timeline, is a toolbar that controls the date. The timeline can be changed for each date by clicking on the respective date button.

The Clear overlaps button would automatically check the timeline and remove any sessions that are overlapping each other. The Removed sessions will be moved to the unscheduled sessions pane at the left.

The Add new micro-location button can be used to instantly add a new room. A modal dialog would open and the micro-location will be instantly added to the timeline once saved.

The Export as iCal allows the organizer to export all the sessions of that event in the popular iCalendar format which can then be imported into various calendar applications.

The Export as PNG saves the entire timeline as a PNG image file. Which can then be printed by the organizers or circulated via other means if necessary.

Core Dependencies

The scheduler makes use of some javascript libraries for the implementation of most of the core functionality

  • Interact.js – For drag-and-drop and resizing
  • Lodash – For array/object manipulations and object cloning
  • jQuery – For DOM Manipulation
  • Moment.js – For date time parsing and calculation
  • Swagger JS – For communicating with our API that is documented according to the swagger specs.
Retrieving data via the API

The swagger js client is used to obtain the sessions data using the API. The client is asynchronously initialized on page load. The client can be accessed from anywhere using the javascript function initializeSwaggerClient.

The swagger initialization function accepts a callback which is called if the client is initialized. If the client is not initialized, the callback is called after that.

var swaggerConfigUrl = window.location.protocol + "//" + window.location.host + "/api/v2/swagger.json";  
window.swagger_loaded = false;  
function initializeSwaggerClient(callback) {  
    if (!window.swagger_loaded) {
        window.api = new SwaggerClient({
            url: swaggerConfigUrl,
            success: function () {
                window.swagger_loaded = true;
                if (callback) {
                    callback();
                }

            }
        });
    } else {
        if (callback) {
            callback();
        }
    }
}

For getting all the sessions of an event, we can do,

initializeSwaggerClient(function () {  
    api.sessions.get_session_list({event_id: eventId}, function (sessionData) {
        var sessions = sessionData.obj;  // Here we have an array of session objects
    });
});

In a similar fashion, all the micro-locations of an event can also be loaded.

Processing the sessions and micro-locations

Each session object is looped through, it’s start time and end time are parsed into moment objects, duration is calculated, and it’s distance from the top in the timeline is calculated in pixels. The new object with additional information, is stored in an in-memory data store, with the date of the event as key, for use in the timeline.

The time configuration is specified in a separate time object.

var time = {  
    start: {
        hours: 0,
        minutes: 0
    },
    end: {
        hours: 23,
        minutes: 59
    },
    unit: {
        minutes: 15,
        pixels: 48,
        count: 0
    },
    format: "YYYY-MM-DD HH:mm:ss"
};

The smallest unit of measurement is 15 minutes and 48px === 15 minutes in the timeline.

Each day of the event is stored in a separate array in the form of Do MMMM YYYY(eg. 2nd May 2013).

The array of micro-location objects is sorted alphabetically by the room name.

Displaying sessions and micro-locations on the timeline

According to the day selected, the sessions for that day are displayed on the timeline. Based on their time, the distance of the session div from the top of the timeline is calculated in pixels and the session box is positioned absolutely. The height of the session in pixels is calculated from it’s duration and set.

For pixels-minutes conversion, the following are used.

/**
 * Convert minutes to pixels based on the time unit configuration
 * @param {number} minutes The minutes that need to be converted to pixels
 * @returns {number} The pixels
 */
function minutesToPixels(minutes) {  
    minutes = Math.abs(minutes);
    return (minutes / time.unit.minutes) * time.unit.pixels;
}

/**
 * Convert pixels to minutes based on the time unit configuration
 * @param {number} pixels The pixels that need to be converted to minutes
 * @returns {number} The minutes
 */
function pixelsToMinutes(pixels) {  
    pixels = Math.abs(pixels);
    return (pixels / time.unit.pixels) * time.unit.minutes;
}
Adding interactivity to the session elements

Interact.js is used to provide interactive capabilities such as drag-drop and resizing.

To know how to use Interact.js, you can checkout some previous blog posts on the same, Interact.js + drag-drop and Interact.js + resizing.

Updating the session information in database on every change

We have to update the session information in database whenever it is moved or resized. Every time a session is moved or resized, a jQuery event is triggered on $(document) along with the session object as the payload.

We listen to this event, and make an API request with the new session object to update the session information in the database.


The scheduler UI is more complex than said in this blog post. To know more about it, you can checkout the scheduler’s javascript code atapp/static/js/admin/event/scheduler.js.

R14 – Memory Quota Exceeded

We, like many other organisations, are using heroku as the deployment server for our project open event organizer server. Things are pretty simple and awesome when your project is in its beginning phase and things run pretty smoothly. But as your project grows, there comes some server problem. And one of the biggest problems as your project grows is memory. Now since various packages have a different amount of memory assigned to you in case of hosting in generic servers such as heroku, so it might result in memory quota exceeded. Recently, we faced such a problem. R14 – Memory Quota Exceeded. Took us quite some time to understand what and why and how this occurred. So let me share a few things I found about this error.

Continue reading R14 – Memory Quota Exceeded

Python code examples

I’ve met many weird examples of  behaviour in python language while working on Open Event project. Today I’d like to share some examples with you. I think this knowledge is necessary, if you’d like to increase a  bit your knowledge in python area.

Simple adding one element to python list:

def foo(value, x=[]):
  x.append(value)
  return x

>>> print(foo(1))
>>> print(foo(2))
>>> print(foo(3, []))
>>> print(foo(4))

OUTPUT

[1] 
[1, 2] 
[3]
[1, 2, 4]

First output is obvious, but second not exactly. Let me explain it, It happens because x(empty list) argument is only evaluated once, So on every call foo(), we modify that list, appending a value to it. Finally we have [1,2, 4] output. I recommend to avoid mutable params as default.

Another example:

Do you know which type it is?

>>> print(type([ el for el in range(10)]))
>>> print(type({ el for el in range(10)}))
>>> print(type(( el for el in range(10))))

Again first and second type are obvious <class ‘list’>, <class ‘set’>. You may  think that last one should return type tuple but it returns a generator <class ‘generator’>.

Example:

Do you think that below code returns an exception?

list= [1,2,3,4,5]
>>> print(list [8:])

If you think that above expression returns index error you’re wrong. It returns empty list [].

Example funny boolean operators

>>> 'c' == ('c' or 'b')
True
>>> 'd' == ('a' or 'd')
False
>>> 'c' == ('c' and 'b')
False 
>>> 'd' == ('a' and 'd')
True

You can think that that OR and AND operators are broken.

You have to know how python interpreter behaves while looking for OR and AND operators.

So OR Expression takes the first statement and checks if it is true. If the first statement is true, then Python returns object’s value without checking second value. If first statement is false interpreter checks second value and returns that value.

AND operator checks if first statement is false, the whole statement has to be false. So it returns first value, but if first statement is true it checks second statement and returns second value.

Below i will show you how it works

>>> 'c' == ('c' or 'b')
>>> 'c' == 'c'
True
>>> 'd' == ('a' or 'd')
>>> 'd' == 'a'
False
>>> 'c' == ('c' and 'b')
>>> 'c' == 'b'
False 
>>> 'd' == ('a' and 'd')
>>> 'd' == 'd'
True

I hope that i have explained you how the python interpreter checks OR and AND operators. So know above examples should be more understandable.

Using ftp-deploy in node.js to publish websites over FTP

In the Open Event Webapp Generator, we recently added the functionality for organisers to submit their ftp credentials and when the website is generated, it’ll automatically upload the website to the chosen ftp server (allowing creation of subdirectory internally, if the organiser so wants).

To achieve we used the very useful nodejs module ftp-deploy which is a wrapper on the popular jsftp library

The code dealing with ftp deployment in our webapp generator can be found here  –

https://github.com/fossasia/open-event-webapp/blob/development/src/backend/ftpdeploy.js

As can be seen, deploying using ftp-deploy is pretty straightforward. Primarily we need a config object

 


  var config = {
    username: ftpDetails.user, //prompted on commandline if not given
    password: ftpDetails.pass, // optional, prompted if none given
    host: ftpDetails.host,
    port: 21,
    localRoot: path.join(__dirname, '/../../dist', appFolder), //local folder containing website
    remoteRoot: ftpDetails.path, //path on ftp server to host website
    exclude: ['.git', '.idea', 'tmp/*'],
    continueOnError: true
};

You can set up some event listeners for events like uploaded uploading and upload-error