Automatic Imports of Events to Open Event from online event sites with Query Server and Event Collect

One goal for the next version of the Open Event project is to allow an automatic import of events from various event listing sites. We will implement this using Open Event Import APIs and two additional modules: Query Server and Event Collect. The idea is to run the modules as micro-services or as stand-alone solutions.

Query Server
The query server is, as the name suggests, a query processor. As we are moving towards an API-centric approach for the server, query-server also has API endpoints (v1). Using this API you can get the data from the server in the mentioned format. The API itself is quite intuitive.

API to get data from query-server

GET /api/v1/search/<search-engine>/query=query&format=format

Sample Response Header

 Cache-Control: no-cache
 Connection: keep-alive
 Content-Length: 1395
 Content-Type: application/xml; charset=utf-8
 Date: Wed, 24 May 2017 08:33:42 GMT
 Server: Werkzeug/0.12.1 Python/2.7.13
 Via: 1.1 vegur

The server is built in Flask. The GitHub repository of the server contains a simple Bootstrap front-end, which is used as a testing ground for results. The query string calls the search engine result scraper scraper.py that is based on the scraper at searss. This scraper takes search engine, presently Google, Bing, DuckDuckGo and Yahoo as additional input and searches on that search engine. The output from the scraper, which can be in XML or in JSON depending on the API parameters is returned, while the search query is stored into MongoDB database with the query string indexing. This is done keeping in mind the capabilities to be added in order to use Kibana analyzing tools.

The frontend prettifies results with the help of PrismJS. The query-server will be used for initial listing of events from different search engines. This will be accessed through the following API.

The query server app can be accessed on heroku.

➢ api/list​: To provide with an initial list of events (titles and links) to be displayed on Open Event search results.

When an event is searched on Open Event, the query is passed on to query-server where a search is made by calling scraper.py with appending some details for better event hunting. Recent developments with Google include their event search feature. In the Google search app, event searches take over when Google detects that a user is looking for an event.

The feed from the scraper is parsed for events inside query server to generate a list containing Event Titles and Links. Each event in this list is then searched for in the database to check if it exists already. We will be using elastic search to achieve fuzzy searching for events in Open Event database as elastic search is planned for the API to be used.

One example of what we wish to achieve by implementing this type of search in the database follows. The user may search for

-Google Cloud Event Delhi
-Google Event, Delhi
-Google Cloud, Delhi
-google cloud delhi
-Google Cloud Onboard Delhi
-Google Delhi Cloud event

All these searches should match with “Google Cloud Onboard Event, Delhi” with good accuracy. After removing duplicates and events which already exist in the database from this list have been deleted, each event is rendered on search frontend of Open Event as a separate event. The user can click on any of these event, which will make a call to event collect.

Event Collect

The event collect project is developed as a separate module which has two parts

● Site specific scrapers
In its present state, event collect has scrapers for eventbrite and ticket-leap which, given a query, scrape eventbrite (and ticket-leap respectively) search results and downloads JSON files of each event using Loklak‘s API.
The scrapers can be developed in any form or any number of scrapers/scraping tools can be added as long as they are in alignment with the Open Event Import API’s data format. Writing tests for these against the concurrent API formats will take care of this. This part will be covered by using a json-validator​ to check against a pre-generated schema.

● REST APIs
The scrapers are exposed through a set of APIs, which will include, but not limited to,
➢ api/fetch-event : ​to scrape any event given the link and compose the data in a predefined JSON format which will be generated based on Open Event Import API. When this function is called on an event link, scrapers are invoked which collect event data such as event, meta, forms etc. This data will be validated against the generated JSON schema. The scraped JSON and directory structure for media files:
➢ api/export : to export all the JSON data containing event information into Open Event Server. As and when the scraping is complete, the data will be added into Open Event’s database as a new event.

How the Import works

The following graphic shows how the import works.




Let’s dive into the workflow. So as the diagram illustrates, the ‘search​’ functionality makes a call to api/list API endpoint provided by query-server which returns with events’ ‘Title’ and ‘Event Link’ from the parsed XML/JSON feed. This list is displayed as Open Event’s search results. Now the results having been displayed, the user can click on any of the events. When the user clicks on any event, the event is searched for in Open Event’s database. Two things happen now:

  • The event page loads if the event is found.
  • If the event does not already exist in the database, clicking on any event will

➢ Insert this event’s title and link in the database and get the event_id

➢ Make a call to api/fetch-event in event-collect which then invokes a site-specific scraper to fetch data about the event the user has chosen

➢ When the data is scraped, it is imported into Open Event database using the previously generated event_id. The page will be loaded using jquery ajax ​as and when the scraping is done.​When the imports are done, the search page refreshes with the new results. The Open Event Orga Server exposes a well documented REST API that can be used by external services to access the data.

Sorting language-translation in Open Event Server project using Jinja 2 dictsort.

Working on the Open Event Server project an issue about arranging language-translation listing in alphabetical order came up. To solve this issue of language listing arrangement i.e. #2817, I found the ‘d0_dictsort’ function in jinja2 to sort dictionaries. It is a defined in jinja2.filters. Python dicts are unsorted and in our web application we at times may want to order them by either their key or value. So this function comes handy.

This is what the function looks like:

do_dictsort(value, case_sensitive=False, by='key')

We can write them in three ways as:

{% for record in my_dictionary|dictsort %}
    case insensitive and sort the dict by key

{% for record in my_dictionary|dicsort(true) %}
    case sensitive and sort the dict by key

{% for record in my_dictionary|dictsort(false, 'value') %}
    sort the dict by value, normally sorted and case insensitive
  1.       The first way is easily understood that dict has been sorted by key not taking case into consideration. It is just in the same way written as dictsort(false).
  2.       Second way is basically the first being case sensitive. dictsort(true) here tells us that case is sensitive.
  3.      Third way is dictsort(false,’value’). The first parameter defines that case insensitive while second parameter defines that it is sorted by ‘value’.

The issues was to sort translation selector for the page in alphabetical order. The languages were stored in a dictionary which to change in order, I found this function very easy and useful.

Basically what we had was:

This is how the function was used in the code for the sort. Like this:

<ul class="dropdown-menu lang-list">
   {% for code in all-languages|dictsort(false,'value') %}
       <li><a  href="#" style="#969191" class="translate" id="{{ code[0] }}">{{  all_languages[code[0]] }}<>a><li>
    {% endfor %}
<ul>


Here:
{{ all_languages }} is the list which contained the languages like French, English, etc., which could be accessed with its global language code. code here(index for all_languages) is a tuple of {‘global_language_code’,’language’} (An example would be (‘fr’,’French’), so code[0] gave me the language_code.

Finally, the result:

This is one of the simple ways to sort your dictionaries.

Open Event Server: No (no-wrap) Ellipsis using jquery!

Yes, the title says it all i.e., Enabling multiple line ellipsis. This was used to solve an issue to keep Session abstract view within 200 characters (#3059) on FOSSASIA‘s Open Event Server project.

There is this one way to ellipsis a paragraph in html-css and that is by using the text-overflow property:

.div_class{
white-space: nowrap;
overflow: hidden;
text-overflow: ellipsis;
}’’

But the downside of this is the one line ellipis. Eg: My name is Medozonuo. I am…..

And here you might pretty much want to ellipsis after a few characters in multiple lines, given that your div space is small and you do want to wrap your paragraph. Or maybe not.

So jquery to the rescue.

There are two ways you can easily do this multiple line ellipsis:

1) Height-Ellipsis (Using the do-while loop):

//script:
if ($('.div_class').height() > 100) {
    var words = $('.div_class').html().split(/\s+/);
    words.push('...');

    do {
        words.splice(-2, 1);
        $('.div_class').html( words.join(' ') );
    } while($('.div_class').height() > 100);
}

Here, you check for the div content’s height and split the paragraph after that certain height and add a “…”, do- while making sure that the paragraphs are in multiple lines and not in one single line. But checkout for that infinite loop.

2) Length-Ellipsis (Using substring function):  

//script:
$.each($('.div_class'), function() {
        if ($(this).html().length > 100) {
               var cropped_words = $(this).html();
               cropped_words = cropped_words.substring(0, 200) + "...";
               $(this).html(cropped_words);
        }
 });

Here, you check for the length/characters rather than the height, take in the substring of the content starting from 0-th character to the 200-th character and then add in extra “…”.

This is exactly how I used it in the code.

$.each($('.short_abstract',function() {
   if ($(this).html().length > 200) {
       var  words = $(this).html();
       words = words.substring(0,200 + "...";
       $(this).html(words);
    }
});


So ellipsing paragraphs over heights and lengths can be done using jQuery likewise.

ember.js – the right choice for the Open Event Front-end

With the development of the API server for the Open Event project we needed to decide which framework to choose for the new Open Event front-end. With the plethora of javascript frameworks available, it got really difficult to decide, which one is actually the right choice. Every month a new framework arrives, and the existing ones keep actively updating themselves often. We decided to go with Ember.js. This article covers the emberJS framework and highlights its advantages over others and  demonstrates its usefulness.

EmberJS is an open-source JavaScript application front end framework for creating web applications, and uses Model-View-Controller (MVC) approach. The framework provides universal data binding. It’s focus lies on scalability.

Why is Ember JS great?

Convention over configuration – It does all the heavy lifting.

Ember JS mandates best practices, enforces naming conventions and generates the boilerplate code for the various components and routes itself. This has advantages other than uniformity. It is easier for other developers to join the project and start working right away, instead of spending hours on existing codebase to understand it, as the core structure of all ember apps is similar. To get an ember app started with the basic route, user doesn’t has to do much, ember does all the heavy lifting.

ember new my-app
ember server

After installing this is all it takes to create your app.

Ember CLI

Similar to Ruby on Rails, ember has a powerful CLI. It can be used to generate boiler plate codes for components, routes, tests and much more. Testing is possible via the CLI as well.

ember generate component my-component
ember generate route my-route
ember test

These are some of the examples which show how easy it is to manage the code via the ember CLI.

Tests.Tests.Tests.

Ember JS makes it incredibly easy to use test-first approach. Integration tests, acceptance tests, and unit tests are in built into the framework. And can be generated from the CLI itself, the documentation on them is well written and it’s really easy to customise them.

ember generate acceptance-test my-test

This is all it takes to set up the entire boiler plate for the test, which you can customise

Excellent documentation and guides

Ember JS has one of the best possible documentations available for a framework. The guides are a breeze to follow. It is highly recommended that, if starting out on ember, make the demo app from the official ember Guides. That should be enough to get familiar with ember.

Ember Guides is all you need to get started.

Ember Data

It sports one of the best implemented API data fetching capabilities. Fetching and using data in your app is a breeze. Ember comes with an inbuilt data management library Ember Data.

To generate a data model via ember CLI , all you have to do is

ember generate model my-model

Where is it being used?

Ember has a huge community and is being used all around. This article focuses on it’s salient features via the example of Open Event Orga Server project of FOSSASIA. The organizer server is primarily based on FLASK with jinja2 being used for rendering templates. At the small scale, it was efficient to have both the front end and backend of the server together, but as it grew larger in size with more refined features it became tough to keep track of all the minor edits and customizations of the front end and the code started to become complex in nature. And that gave birth to the new project Open Event Front End which is based on ember JS which will be covered in the next week.

With the orga server being converted into a fully functional API, the back end and the front end will be decoupled thereby making the code much cleaner and easy to understand for the other developers that may wish to contribute in the future. Also, since the new front end is being designed with ember JS, it’s UI will have a lot of enhanced features and enforcing uniformity across the design would be much easier with the help of components in ember. For instance, instead of making multiple copies of the same code, components are used to avoid repetition and ensure uniformity (change in one place will reflect everywhere)

<.div class="{{if isWide 'event wide ui grid row'}}">
  {{#if isWide}}
    {{#unless device.isMobile}}
      <.div class="ui card three wide computer six wide tablet column">
        <.a class="image" href="{{href-to 'public' event.identifier}}">
          {{widgets/safe-image src=(if event.large event.large event.placeholderUrl)}}
        <./a>
      <./div>
    {{/unless}}
  {{/if}}
  <.div class="ui card {{unless isWide 'event fluid' 'thirteen wide computer ten wide tablet sixteen wide mobile column'}}">
    {{#unless isWide}}
      <.a class="image" href="{{href-to 'public' event.identifier}}">
        {{widgets/safe-image src=(if event.large event.large event.placeholderUrl)}}
      <./a>
    {{/unless}}
    <.div class="main content">
      <.a class="header" href="{{href-to 'public' event.identifier}}">
        <.span>{{event.name}}<./span>
      <./a>
      <.div class="meta">
        <.span class="date">
          {{moment-format event.startTime 'ddd, MMM DD HH:mm A'}}
        <./span>
      <./div>
      <.div class="description">
        {{event.shortLocationName}}
      <./div>
    <./div>
    <.div class="extra content small text">
      <.span class="right floated">
        <.i role="button" class="share alternate link icon" {{action shareEvent event}}><./i>
      <./span>
      <.span>
        {{#if isYield}}
          {{yield}}
        {{else}}
          {{#each tags as |tag|}}
            <.a>{{tag}}<./a>
          {{/each}}
        {{/if}}
      <./span>
    <./div>
  <./div>
<./div>

This is a perfect example of the power of components in ember, this is a component for event information display in a card format which in addition to being rendered differently for various screen sizes can act differently based on passed parameters, thereby reducing the redundancy of writing separate components for the same.

Ember is a step forward towards the future of the web. With the help of Babel.js it is possible to write ES6/2015 syntax and not worry about it’s compatibility with the browsers. It will take care of it.

This is perfectly valid and will be compatible with majority of the supported browsers.

actions: {
  submit() {
    this.onValid(()=> {
    });
  }
}

 

Some references used for the blog article:

  1. https://www.codeschool.com/blog/2015/10/26/7-reasons-to-use-ember-js/
  2. https://www.quora.com/What-are-the-advantages-of-using-Ember-js
  3. Official Ember Guides: https://guides.emberjs.com

 
This page/product/etc is unaffiliated with the Ember project. Ember is a trademark of Tilde Inc

Exporting Speakers and Sessions list download as CSV in the Open Event Server

In an event management system there is a need for organizers to get speakers data from the system and be able to use and process it elsewhere. Organizers might need a list of emails, phone numbers or other information. An “export” of speakers and sessions in a CSV file that can be opened in any spreadsheet application is a good way to obtain this data. Therefore we implemented an export as CSV funtion in the Open Event Server.

 

Now the Open Event Orga Server allows event organizers to export a list of speakers and details about their sessions such as Session status, Session title, Session speaker, Session track.

The speaker csv includes details about the speaker such as Name, Sessions, Email, Status, Mobile, Organisation, and Position.

 

How did we implement it:

When clicking on the Export As CSV button on either of the pages it calls the download_speakers_as_csv / download_sessions_as_csv  from speakers / sessions tab .

The Speaker’s Tab

Session’s Tab

The functions are defined in sessions.py file.

First we get data from the sessions table by event_id and the event details from event table :

sessions = DataGetter.get_sessions_by_event_id(event_id)

# This is for getting event name to put as the filename

event = DataGetter.get_event(event_id)

 

Then we go on with creating the csv file.

We iterate through all the sessions and find the speaker associated with each. We save each row in a new list named data and after each iteration add it to a main list .

main = [["Session Title", "Session Speakers", \
         "Session Track", "Session Abstract", "Email Sent"]]
for session in sessions:
    if not session.deleted_at:
        data = [session.title + " (" + session.state + ")" if session.title else '']
        if session.speakers:
            inSession = ''
            for speaker in session.speakers:
                if speaker.name:
                    inSession += (speaker.name + ', ')
            data.append(inSession[:-2])
        data.append(session.track.name if session.track.name else '')
        data.append(strip_tags(session.short_abstract) if session.short_abstract else '')
        data.append('Yes' if session.state_email_sent else 'No')
        main.append(data)

In the last part of the code python’s csv  module is used to create a csv file out of the nested lists. The csv module takes care of properly escaping the characters such as punctuation marks ( like comma ) and appending a newline character in the end of each list .

Snippet of Code from sessions.py file

 

In the last few lines of this code section , you can see the headers being added , necessary for downloading the file on the user’s end.

The make_response function is imported from flask package.

make_response : Converts the return value from a view function to a real response object  ( as documented here).

 

The  exported filename format is like :

‘%event-name%-Speakers.csv’ , ‘%event-name%-Sessions.csv

Thus getting the list of speakers and session details as csv files.

Speaker’s CSV

Session’s CSV

 

Ticket Ordering or Positioning (back-end)

One of the many feature requests that we got for our open event organizer server or the eventyay website is ticket ordering. The event organizers wanted to show the tickets in a particular order in the website and wanted to control the ordering of the ticket. This was a common request by many and also an important enhancement. There were two main things to deal with when ticket ordering was concerned. Firstly, how do we store the position of the ticket in the set of tickets. Secondly, we needed to give an UI in the event creation/edit wizard to control the order or position of a ticket. In this blog, I will talk about how we store the position of the tickets in the backend and use it to show in our public page of the event.

Continue reading Ticket Ordering or Positioning (back-end)

The Open Event Ecosystem

{ Repost from my personal blog @ https://blog.codezero.xyz/open-event-ecosystem }

This post contains data collected and compiled from various sources that include (but not limited to) project readme files, my presentation from FOSSASIA 17, wikipedia, my head.

This aims to be a place for any new contributor to know the basics about the entire Open Event Project.

This could also help newcomers choose an Open Event sub-project of their liking and start contributing .

The Open Event Project offers event managers a platform to organise all kinds of events including concerts, conferences, summits and regular meet-ups. The components support organisers in all stages from event planning to publishing, marketing and ticket sales. Automated web and mobile apps help attendees to get information easily.

There are seven components of the project:

  • Open Event Universal Format – A standard JSON Schema specification for data sharing/transfer/import/export between various components of Open Event.
  • Open Event API Server – The core of the project. The API Server and database.
  • Open Event Frontend – The primary frontend for the Open Event API Server where organisers, speakers and attendees can sign-up and perform various functions.
  • Open Event Organiser App – A cross-platform mobile application for Organisers with ticketing, checking and quick management capabilities.
  • Open Event Android application generator – An android application generator that allows event organisers to generate customised android applications for their events.
  • Open Event Web application generator – A web application generator that allows event organisers to generate customised static easily-hostable web pages for their events.
  • Open Event Data scrappers – A scrapper that allows users to scrape events from popular websites such as eventbrite into the Open Event Universal Format for it to be imported into Open Event.

Open Event Universal Format

A standard JSON Schema specification for data sharing/transfer/import/export between various components of Open Event.

Repository: fossasia/open-event.

Open Event API Server

The core of the project. The API Server and database.

Repository: fossasia/open-event-orga-server.

The system exposes a RESTful API to fetch and modify data. It also provides endpoints that allow organisers to import and export event data in a standard compressed file format that includes the event data in JSON (Open Event Universal Format) and binary media files like images and audio.

The Open Event API Server comprises of:

  • Flask web framework – Flask is a microframework for python to create web applications. Flask also provided us with a Jinja2 templating engine.
  • PostgreSQL – Our database. PostgreSQL is an open-sourced Object-relational database management system (ORDBMS). We use SQLAlchemy ORM to communicate with the database and perform database operations.
  • Celery – Celery is a Distributed Task Queue. It allows us to run time consuming and/or resource intensive tasks in the background (or) on a separate worker server. We use celery to process event import/export tasks to process email send requests.
  • Redis – Redis is an in-memory data structure store. It’s generally used for caching and for message brokering b/w different services due it’s insanely fast read-write speeds (since it’s an in-memory data store). We use it for caching results of time-consuming less volatile database calls and also for brokering jobs and their statuses b/w the web server and Celery task queue.

In the near future, we plan to implement more additional components too.

  • Elasticsearch – a distributed, RESTful search and analytics engine. To start with, we’ll be using it to index our events and provide much fast search results to the user.
  • Kibana – data visualization and analytics plugin for elasticsearch. It will allow us to better understand search trends, user demographics and help us provide a better user experience.

We’re now in the process of slowly phasing out the Open Event Server’s existing UI and keep it only as an API Server since we’re moving towards an API-Centric approach with a Fresh new Open Event Frontend.

The Open Event server’s repository contains multiple branches each serving a specific purpose.

  • development – All development goes on in this branch. If you’re making a contribution, please make a pull request to development. PRs to must pass a build check and a unit-test check on Travis (https://open-event-dev.herokuapp.com – Is running off the development branch. It is hosted on Heroku.). This is probably the most unstable branch of all.
  • master – This contains shipped code. After significant features/bug-fixes are accumulated on development, we make a version update, and make a release. (https://eventyay.com – Is running off the master branch. (whichever is the latest release.) Hosted on Google Cloud Platform (Google Container Engine + Kubernetes).
  • staging – This branch is mainly for testing eventyay server configurations on a staging server without affecting the production environment. Push access is restricted to the devops team.
  • gh-pages – This contains the documentation website on http://dev.eventyay.com. The site is build automatically on each commit in the development branch through a script and using travis. It includes the md files of the Readme and /docs folder.

The Open Event Server has Unit tests to ensure each part of the server works as expected. This is also part of our workflow. So, every PR made to the repository, is tested by running the Unit tests and the code coverage is reported on the PR itself. (We use travis and codecov for continuous integration and code coverage reporting respectively).

Open Event Frontend

The primary frontend for the Open Event API Server where organisers, speakers and attendees can sign-up and perform various functions.

Repository: fossasia/open-event-frontend.

Open Event frontend is built on Ember.js – A javascript web application framework. Ember.js uses Ember data – its data persistence module to communicate with the Open Event API Server via the exposed RESTful endpoints.

The frontend is built upon components that are re-usable across various parts of the application. This reduces code duplication and increases maintainability.

The frontend also has various services to share data and functionality b/w components/routes/controllers.

Each merge into the Open Event frontend repository triggers two deployments:

Currently, both the staging and development use the development branch since the frontend is still under active development. Once we reach the release stage, staging & production deployments will start using the master branch just like the Open Event API Server.

As a part of the development workflow, to ensure proper code-styles throughout the project, and to prevent regressions, the project has Acceptance tests, Integration tests, Unit tests and lint checks on both javascript (*.js) and handlebar files (*.hbs). These tests are run for every PR made to the repository. (We use travis and codecov for continuous integration and code coverage reporting respectively).

Open Event Organizer App

A cross-platform mobile application for Organiser with ticketing, checking and quick management capabilities.

Repository: fossasia/open-event-orga-app.

The organiser application is a mobile application that can run on Android, iOS and Windows Phone. It is built using Ionic Framework – cross-platform app development framework. The app uses the RESTful APIs exposed by the Open Event API Server to get data and perform operations.

The core features of this application are

  • Scan a QR code from an attendee’s ticket to view information about the attendee and to check him/her in.
  • Check-in attendees (Attendees can be searched for using name and/or email)
  • Continuous data sync with the Open Event Organiser Server

Other planned feature include:

  • Overview of sales – The organisers can see how their event is performing with just a few taps
  • Overview of tracks and sessions – They can see what sessions are next and can use that information to (for example) get everything ready for that session.
  • Quick session re-scheduling – They can re-schedule sessions if required. This should also trigger notification to participant that have registered for that event.
  • Push notifications for certain triggers – (for example) Organisers can get notifications if any session proposal is received.

This project has only one development branch (master). Each commit to that branch triggers an apk deployment to the apk branch via Travis.

Open Event Android Application Generator

An android application generator that allows event organisers to generate customised android applications for their events.

Repository: fossasia/open-event-android.

This project consists of two components.

  • The App Generator – A web application that is hosted on a server and generates an event Android app from a zip with JSON and binary files (examples here) or through an API.
  • A generic Android app – the output of the app generator. The mobile app can be installed on any Android device for browsing information about the event. Updates can be made automatically through API endpoint connections from an online source (e.g. server), which needs to defined in the provided event zip with the JSON files. The Android app has a standard configuration file, that sets the details of the app (e.g. color scheme, logo of event, link to JSON app data).

  • Flask web framework – Flask is a microframework for python to create web applications. Flask also provided us with a Jinja2 templating engine.
  • Celery – Celery is a Distributed Task Queue. It allows us to run time consuming and/or resource intensive tasks in the background (or) on a separate worker server. The android application compile and build process is run as a celery job. This also allows multiple users to simultaneously use the generator.
  • Redis – Redis is an in-memory data structure store. It’s generally used for caching and for message brokering b/w different services due it’s insanely fast read-write speeds (since it’s an in-memory data store). We use it for brokering jobs and their statuses b/w the web server and Celery task queue.
  • Java Development Kit (JDK) – Java Development Kit provides us with a set of tools consisting of (but not limited to) a compiler, runtime environment, loader which enables us to compiler, build and run java based applications.
  • Android SDK Tools – The Android SDK Toolset provides us with Android API libraries, debugging tools, emulation capabilities and other tools that are needed to develop, build and run java based android applications.
  • Gradle Build Tool – Gradle is an open source build automation system. It allows developers to define a build process as a set of tasks that can be easily executed on any machine with predictable outputs as long as the gradle files are available.

As with other projects, this also has multiple branches each serving a different purpose

  • development – All development goes on in this branch. If you’re making a contribution, you are supposed to make a pull request to development. PRs to master must pass a build check and a unit-test (app/src/test) check on Travis.
  • master – This contains shipped code. After significant features/bugfixes are accumulated on development, we make a version update, and make a release. All tagged commits on master branch will automatically generate a release on Github with a copy of fDroid-debug and GooglePlay-debug apks.
  • apk – This branch contains two apk’s, that are automatically generated on merged pull request a) from the dev branch and b) from the master branch using the Open Event sample of the FOSSASIA Summit. This branch also assists in deploying the generator to http://droidgen.eventyay.com by triggering a travis build every time an apk is pushed to this branch. The reason this type of a round-about way was implemented is that, travis doesn’t allow android and docker builds on the same branch. So, we’re forced to use the apk branch to do the docker build for us.

Open Event Web application generator

A web application generator that allows event organisers to generate customised static easily-hostable web pages for their events.

Repository: fossasia/open-event-webapp.

The Open Event Web App project has two components :

  • An event website generator – A web application that generates the event website based on event data provided by the user either in the form of an archive as per the Open Event Universal Format or an API Endpoint to an Open Event API Server instance
  • A generic web application – This will be customised and additional pages will be generated from a set of templates by the generator based on the provided event data. The generated website bundle can be hosted on any static hosting provider (Github pages, any simple web hosting service, or any web server that can serve HTML files etc).

The generator also gives you the ability to directly upload the generated files to any server (via FTP) or to the gh-pages branch of any repository on GitHub.

  • Express.js – A web application framework for Node.js. The web application generator’s user-facing frontend runs on Express.js framework.
  • Socket.IO – A javascript library for real-time web applications. It allows a client and a server to communicate with each other simultaneously in a real-time manner. (Confusing ? If you wanna build something like a chat-room … you’d need something like Socket.IO). The web application generator uses Socket.IO to handle multiple clients and also to show the correct progress of the website generation to each of those clients in a real-time manner.
  • The web generator also uses SASS – which adds awesome features (Superpowers as their site says) to good-old CS, and Handlebars – which is a templating engine that let’s you easily generate HTML output based on a templates from a given javascript object.

The branches are similar to other projects under Open Event.

  • development – All development goes on in this branch. If you’re making a contribution, you are supposed to make a pull request to development. PRs to master must pass a build check and a unit-test (test/serverTest.js) check on Travis. Gets deployed automatically to https://opev-webgen-dev.herokuapp.com .
  • master – This contains shipped code. After significant features/bugfixes are accumulated on development, we make a version update, and make a release.

Open Event Data Scrappers

A scrapper that allows users to scrape events from popular websites such as eventbrite into the Open Event Universal Format for it to be imported into Open Event.

Repository: fossasia/event-collect.

As of now, only eventbrite is supported for data scrapping. This allows new users of Open Event to import their events from other ticketing/event management websites very easily. The scrapper accepts a query from the user, and gets the event page for that query on eventbrite, parses the HTML DOM to get the required data and compiles everything into the Open Event Universal Format so that it could be imported into Open Event.

The scrapper is written in Python and uses Beautiful Soup to parse the DOM.

Future plans include combining fossasia/query-server, Open Event server and event collect to enable automatic import of events from other websites based on user searches on eventyay.


The Open Event Ecosystem is vast and has plenty of contribution opportunities to both beginners and experts alike. Starting with contributions is as easy as picking a project, choosing an issue and submitting a PR. No strings attached. (Oh wait. there is just one string attached. Ensure you read & follow the “Best Practices at FOSSASIA Guide”)

Generating xCal calendar in python

{ Repost from my personal blog @ https://blog.codezero.xyz/generate-xcal-calendar-in-python }

“xCal”, is an XML format for iCalendar data.

The iCalendar data format (RFC5545) is a widely deployed interchange format for calendaring and scheduling data.

A Sample xCal document

<?xml version="1.0" encoding="utf-8"?>  
<iCalendar xmlns:xCal="urn:ietf:params:xml:ns:xcal">  
    <vcalendar>
        <version>2.0</version>
        <prodid>-//Pentabarf//Schedule 1.0//EN</prodid>
        <x-wr-caldesc>FOSDEM 2016</x-wr-caldesc>
        <x-wr-calname>Schedule for events at FOSDEM 2016</x-wr-calname>
        <vevent>
            <method>PUBLISH</method>
            <uid>123e4567-e89b-12d3-a456-426655440000</uid>
            <dtstart>20160131T090000</dtstart>
            <dtend>20160131T091000</dtend>
            <duration>00:10:00:00</duration>
            <summary>Introduction to the SDR Track- Speakers, Topics, Algorithm</summary>
            <description>&lt;p&gt;The opening talk for the SDR devroom at FOSDEM 2016.&lt;/p&gt;</description>
            <class>PUBLIC</class>
            <status>CONFIRMED</status>
            <categories>Software Defined Radio</categories>
            <url>https:/fosdem.org/2016/schedule/event/sdrintro/</url>
            <location>AW1.125</location>
            <attendee>Martin Braun</attendee>
        </vevent>
    </vcalendar>
</iCalendar>

Each event/session will be in a seperate vevent block. Each speaker/attendee of an event/session will be in an attendee block inside a vevent block.

Some important elements are:

  1. version – Has the version of the iCalendar data
  2. prodid – Contains the name of the application/generator that generated this document
  3. x-wr-caldesc – A descriptive name for this calendar
  4. x-wr-calname – A description of the calendar

The structure and keywords used in xCal are the same as those used in the iCal format. To generate the XML document, we’ll be using python’s ElementTreeXML API that is part of the Python standard library.

We’ll be using two main classes of the ElementTree API:

  1. Element – used to create a standard node. (Used for the root node)
  2. SubElement – used to create a sub element and attache the new node to a parent

Let’s start with the root iCalendar node and set the required attributes.

from xml.etree.ElementTree import Element, SubElement, tostring

i_calendar_node = Element('iCalendar')  
i_calendar_node.set('xmlns:xCal', 'urn:ietf:params:xml:ns:xcal')

Now, to add the vcalendar node to the iCalendar node.

v_calendar_node = SubElement(i_calendar_node, 'vcalendar')

Let’s add the other aspects of the calendar to the vcalendar node as separate sub nodes.

version_node = SubElement(v_calendar_node, 'version')  
version_node.text = '2.0'

prod_id_node = SubElement(v_calendar_node, 'prodid')  
prod_id_node.text = '-//fossasia//open-event//EN'

cal_desc_node = SubElement(v_calendar_node, 'x-wr-caldesc')  
cal_desc_node.text = "Calendar"

cal_name_node = SubElement(v_calendar_node, 'x-wr-calname')  
cal_name_node.text = "Schedule for sessions"

Now, we have added information about our calendar. Now to add the actual events to the calendar. Each event would be a vevent node, a child of vcalendar node. We can loop through all our available event/sessions and add them to the calendar.

for session in sessions:  
    v_event_node = SubElement(v_calendar_node, 'vevent')

    uid_node = SubElement(v_event_node, 'uid')
    uid_node.text = str(session.id)

    dtstart_node = SubElement(v_event_node, 'dtstart')
    dtstart_node.text = session.start_time.isoformat()

    dtend_node = SubElement(v_event_node, 'dtend')
    dtend_node.text = tz.localize(session.end_time).isoformat()

    duration_node = SubElement(v_event_node, 'duration')
    duration_node.text =  "00:30"

    summary_node = SubElement(v_event_node, 'summary')
    summary_node.text = session.title

    description_node = SubElement(v_event_node, 'description')
    description_node.text = session.short_abstract

    class_node = SubElement(v_event_node, 'class')
    class_node.text = 'PUBLIC'

    status_node = SubElement(v_event_node, 'status')
    status_node.text = 'CONFIRMED'

    categories_node = SubElement(v_event_node, 'categories')
    categories_node.text = session.session_type.name

    url_node = SubElement(v_event_node, 'url')
    url_node.text = "https://some.conf/event/" + str(session.id)

    location_node = SubElement(v_event_node, 'location')
    location_node.text = session.microlocation.name

    for speaker in session.speakers:
        attendee_node = SubElement(v_event_node, 'attendee')
        attendee_node.text = speaker.name

Please note that all the timings in the XML Document must comply with ISO 8601 and must have the date+time+timezone. Example: 2007-04-05T12:30-02:00.

We’re still not done yet. We now have the XML document as an Element object. But we’ll be needing it as a string to either store it somewhere or display it.

The document can be converted to a string by using the ElementTree API’s tostring helper method and passing the root node.

xml_as_string = tostring(i_calendar_node)

And that’s it. You now have a proper XML document representing your events.

KISS Datatable

Recenlty I’ve faced a problem with sorting columns in Datatable.

What is Datatable?

Datatable is a plug-in for Jquery library. It provides a lot of features like pagination, quick search or multi-column ordering. Besides, you can develop easily Bootstrap or Foundation ui css styles. There are also more other option but It doesn’t make sense to list it here, because you can visit their site and you can read clearly documentation. On Datatable website you can see a lot of examples. First of them shows how to improve your ordinary table to awesome and rich of features table. One function changes everything, It’s fantastic!  

$('#myTable').DataTable();

Returning to my problem which I’ve faced, as I told it was problem related to sorting column in table.

I know sorting is a trivial thing. I hope that everyone knows it 🙂 Sorting by a date is also implemented in a datatable library. So everything is clear when we don’t change date format to human readable format. I mean something like this ‘3 hours ago’, ‘1 year ago’.

When Open Event team tested how datatable manages ordering columns in that format it didn’t work. It’s quite hard to sort by that format, So I’ve invented an idea. Surely you are wondering what i’ve invented. I’ve postponed my minds about sort by this values. It can direct to overwork. When I thought about it, weird ideas came to my mind, a lots of conditions, If statements… Therefore I’ve resigned from this. I’ve used KISS principle. KISS means ‘keep it simple stupid’. I like it!

Therefore that sorting is implemented on frontend side. I’ve decided not to display human readable date format at the beginning. I find  all dates which have format “YYYY-MM-DD HH:mm” then I replace that format to human readable format. So it’s very quick and comfortable, and doesn’t require a lot conditions to write. Of course I’ve tried to implement it in Datatable library. I suppose that it would  require more effort than it’s now.

Below You can see great function which changes a date in frontend side but does not change data in a datatable. So sorting process takes place in a datatable using format  “YYYY-MM-DD HH:mm” but user see human readable format. Isn’t it awesome?!

function change_column_time_to_humanize_format(datatable_name, column_id) {
  $(datatable_name).each(function( key, value ) {
    $(value).children().each(function( key1, value2 ) {
       if(key1 === column_id ){
          var current_value = $(value2).text().slice(0, 16);
          var changed_value = moment(current_value, "YYYY-MM-DD hh:mm").fromNow()
          var isValid = moment(current_value, "YYYY-MM-DD HH:mm", true).isValid()
          if (changed_value !== current_value && isValid === true){
              $(value2).text(changed_value)
          }
      }
  });
});

 

Accepting Stripe payments on behalf of a third-party

{ Repost from my personal blog @ https://blog.codezero.xyz/accepting-stripe-payments-on-behalf-of-a-third-party }

In Open Event, we allow the organizer of each event to link their Stripe account, so that all ticket payments go directly into their account. To make it simpler for the organizer to setup the link, we have a Connect with stripe button on the event creation form.

Clicking on the button, the organizer is greeted with a signup flow similar to Login with Facebook or any other social login. Through this process, we’re able to securely and easily obtain the credentials required to accept payments on behalf of the organizer.

For this very purpose, stripe provides us with an OAuth interface called as Stripe Connect. Stripe Connect allows us to connect and interact with other stripe accounts through an API.

We’ll be using Python’s requests library for making all the HTTP Requests to the API.
You will be needing a stripe account for this.

Registering your platform
The OAuth Flow

The OAuth flow is similar to most platforms.

  • The user is redirected to an authorization page where they login to their stripe account and authorize your app to access their account
  • The user is then redirected back to a callback URL with an Authorization code
  • The server makes a request to the Token API with the Authorization code to retrieve the access_token, refresh_token and other credentials.

Implementing the flow

Redirect the user to the Authorization URL.
https://connect.stripe.com/oauth/authorize?response_type=code&client_id=ca_8x1ebxrl8eOwOSqRTVLUJkWtcfP92YJE&scope=read_write&redirect_uri=http://localhost/stripe/callback  

The authorization url accepts the following parameters.

  1. client_id – The client ID acquired when registering your platform.required.
  2. response_type – Response type. The value is always code. required.
  3. redirect_uri – The URL to redirect the customer to after authorization.
  4. scope – Can be read_write or read_only. The default is read_only. For analytics purposes, read_only is appropriate; To perform charges on behalf of the connected user, We will need to request read_write scope instead.

The user will be taken to stripe authorization page, where the user can login to an existing account or create a new account without breaking the flow. Once the user has authorized the application, he/she is taken back to the Callback URL with the result.

Requesting the access token with the authorization code

The user is redirected back to the callback URL.

If the authorization failed, the callback URL has a query string parameter error with the error name and a parameter error_description with the description of the error.

If the authorization was a success, the callback URL has the authorization code in the code query string parameter.

import requests

data = {  
    'client_secret': 'CLIENT_SECRET',
    'code': 'AUTHORIZATION_CODE',
    'grant_type': 'authorization_code'
}

response = requests.post('https://connect.stripe.com/oauth/token', data=data)

The client_secret is also obtained when registering your platform. The codeparameter is the authorization code.

On making this request, a json response will be returned.

If the request was a success, the following response will be obtained.

{
  "token_type": "bearer",
  "stripe_publishable_key": PUBLISHABLE_KEY,
  "scope": "read_write",
  "livemode": false,
  "stripe_user_id": USER_ID,
  "refresh_token": REFRESH_TOKEN,
  "access_token": ACCESS_TOKEN
}

If the request failed for some reason, an error will be returned.

{
  "error": "invalid_grant",
  "error_description": "Authorization code does not exist: AUTHORIZATION_CODE"
}

The access_token token obtained can be used as the secret key to accept payments like discussed in Integrating Stripe in the Flask web framework.