The Open Event Ecosystem

This post contains data collected and compiled from various sources that include (but not limited to) project readme files, my presentation from FOSSASIA 17, Wikipedia, my head.

This aims to be a place for any new contributor to know the basics about the entire Open Event Project.

This could also help newcomers choose an Open Event sub-project of their liking and start contributing .

The Open Event Project offers event managers a platform to organise all kinds of events including concerts, conferences, summits and regular meet-ups. The components support organisers in all stages from event planning to publishing, marketing and ticket sales. Automated web and mobile apps help attendees to get information easily.

There are seven components of the project:

  • Open Event Universal Format – A standard JSON Schema specification for data sharing/transfer/import/export between various components of Open Event.
  • Open Event API Server – The core of the project. The API Server and database.
  • Open Event Frontend – The primary frontend for the Open Event API Server where organisers, speakers and attendees can sign-up and perform various functions.
  • Open Event Organiser App – A cross-platform mobile application for Organisers with ticketing, checking and quick management capabilities.
  • Open Event Android application generator – An android application generator that allows event organisers to generate customised android applications for their events.
  • Open Event Web application generator – A web application generator that allows event organisers to generate customised static easily-hostable web pages for their events.
  • Open Event Data scrappers – A scrapper that allows users to scrape events from popular websites such as eventbrite into the Open Event Universal Format for it to be imported into Open Event.

Open Event Universal Format

A standard JSON Schema specification for data sharing/transfer/import/export between various components of Open Event.

Repository: fossasia/open-event.

Open Event API Server

The core of the project. The API Server and database.

Repository: fossasia/open-event-orga-server.

The system exposes a RESTful API to fetch and modify data. It also provides endpoints that allow organisers to import and export event data in a standard compressed file format that includes the event data in JSON (Open Event Universal Format) and binary media files like images and audio.

The Open Event API Server comprises of:

  • Flask web framework – Flask is a microframework for python to create web applications. Flask also provided us with a Jinja2 templating engine.
  • PostgreSQL – Our database. PostgreSQL is an open-sourced Object-relational database management system (ORDBMS). We use SQLAlchemy ORM to communicate with the database and perform database operations.
  • Celery – Celery is a Distributed Task Queue. It allows us to run time consuming and/or resource intensive tasks in the background (or) on a separate worker server. We use celery to process event import/export tasks to process email send requests.
  • Redis – Redis is an in-memory data structure store. It’s generally used for caching and for message brokering b/w different services due it’s insanely fast read-write speeds (since it’s an in-memory data store). We use it for caching results of time-consuming less volatile database calls and also for brokering jobs and their statuses b/w the web server and Celery task queue.

In the near future, we plan to implement more additional components too.

  • Elasticsearch – a distributed, RESTful search and analytics engine. To start with, we’ll be using it to index our events and provide much fast search results to the user.
  • Kibana – data visualization and analytics plugin for elasticsearch. It will allow us to better understand search trends, user demographics and help us provide a better user experience.

We’re now in the process of slowly phasing out the Open Event Server’s existing UI and keep it only as an API Server since we’re moving towards an API-Centric approach with a Fresh new Open Event Frontend.

The Open Event server’s repository contains multiple branches each serving a specific purpose.

  • development – All development goes on in this branch. If you’re making a contribution, please make a pull request to development. PRs to must pass a build check and a unit-test check on Travis (https://open-event-dev.herokuapp.com – Is running off the development branch. It is hosted on Heroku.). This is probably the most unstable branch of all.
  • master – This contains shipped code. After significant features/bug-fixes are accumulated on development, we make a version update, and make a release. (https://eventyay.com – Is running off the master branch. (whichever is the latest release.) Hosted on Google Cloud Platform (Google Container Engine + Kubernetes).
  • staging – This branch is mainly for testing eventyay server configurations on a staging server without affecting the production environment. Push access is restricted to the devops team.
  • gh-pages – This contains the documentation website on http://dev.eventyay.com. The site is build automatically on each commit in the development branch through a script and using travis. It includes the md files of the Readme and /docs folder.

The Open Event Server has Unit tests to ensure each part of the server works as expected. This is also part of our workflow. So, every PR made to the repository, is tested by running the Unit tests and the code coverage is reported on the PR itself. (We use travis and codecov for continuous integration and code coverage reporting respectively).

Open Event Frontend

The primary frontend for the Open Event API Server where organisers, speakers and attendees can sign-up and perform various functions.

Repository: fossasia/open-event-frontend.

Open Event frontend is built on Ember.js – A javascript web application framework. Ember.js uses Ember data – its data persistence module to communicate with the Open Event API Server via the exposed RESTful endpoints.

The frontend is built upon components that are re-usable across various parts of the application. This reduces code duplication and increases maintainability.

The frontend also has various services to share data and functionality b/w components/routes/controllers.

Each merge into the Open Event frontend repository triggers two deployments:

Currently, both the staging and development use the development branch since the frontend is still under active development. Once we reach the release stage, staging & production deployments will start using the master branch just like the Open Event API Server.

As a part of the development workflow, to ensure proper code-styles throughout the project, and to prevent regressions, the project has Acceptance tests, Integration tests, Unit tests and lint checks on both javascript (*.js) and handlebar files (*.hbs). These tests are run for every PR made to the repository. (We use travis and codecov for continuous integration and code coverage reporting respectively).

Open Event Organizer App

A cross-platform mobile application for Organiser with ticketing, checking and quick management capabilities.

Repository: fossasia/open-event-orga-app.

The organiser application is a mobile application that can run on Android, iOS and Windows Phone. It is built using Ionic Framework – cross-platform app development framework. The app uses the RESTful APIs exposed by the Open Event API Server to get data and perform operations.

The core features of this application are

  • Scan a QR code from an attendee’s ticket to view information about the attendee and to check him/her in.
  • Check-in attendees (Attendees can be searched for using name and/or email)
  • Continuous data sync with the Open Event Organiser Server

Other planned feature include:

  • Overview of sales – The organisers can see how their event is performing with just a few taps
  • Overview of tracks and sessions – They can see what sessions are next and can use that information to (for example) get everything ready for that session.
  • Quick session re-scheduling – They can re-schedule sessions if required. This should also trigger notification to participant that have registered for that event.
  • Push notifications for certain triggers – (for example) Organisers can get notifications if any session proposal is received.

This project has only one development branch (master). Each commit to that branch triggers an apk deployment to the apk branch via Travis.

Open Event Android Application Generator

An android application generator that allows event organisers to generate customised android applications for their events.

Repository: fossasia/open-event-android.

This project consists of two components.

  • The App Generator – A web application that is hosted on a server and generates an event Android app from a zip with JSON and binary files (examples here) or through an API.
  • A generic Android app – the output of the app generator. The mobile app can be installed on any Android device for browsing information about the event. Updates can be made automatically through API endpoint connections from an online source (e.g. server), which needs to defined in the provided event zip with the JSON files. The Android app has a standard configuration file, that sets the details of the app (e.g. color scheme, logo of event, link to JSON app data).

  • Flask web framework – Flask is a microframework for python to create web applications. Flask also provided us with a Jinja2 templating engine.
  • Celery – Celery is a Distributed Task Queue. It allows us to run time consuming and/or resource intensive tasks in the background (or) on a separate worker server. The android application compile and build process is run as a celery job. This also allows multiple users to simultaneously use the generator.
  • Redis – Redis is an in-memory data structure store. It’s generally used for caching and for message brokering b/w different services due it’s insanely fast read-write speeds (since it’s an in-memory data store). We use it for brokering jobs and their statuses b/w the web server and Celery task queue.
  • Java Development Kit (JDK) – Java Development Kit provides us with a set of tools consisting of (but not limited to) a compiler, runtime environment, loader which enables us to compiler, build and run java based applications.
  • Android SDK Tools – The Android SDK Toolset provides us with Android API libraries, debugging tools, emulation capabilities and other tools that are needed to develop, build and run java based android applications.
  • Gradle Build Tool – Gradle is an open source build automation system. It allows developers to define a build process as a set of tasks that can be easily executed on any machine with predictable outputs as long as the gradle files are available.

As with other projects, this also has multiple branches each serving a different purpose

  • development – All development goes on in this branch. If you’re making a contribution, you are supposed to make a pull request to development. PRs to master must pass a build check and a unit-test (app/src/test) check on Travis.
  • master – This contains shipped code. After significant features/bugfixes are accumulated on development, we make a version update, and make a release. All tagged commits on master branch will automatically generate a release on Github with a copy of fDroid-debug and GooglePlay-debug apks.
  • apk – This branch contains two apk’s, that are automatically generated on merged pull request a) from the dev branch and b) from the master branch using the Open Event sample of the FOSSASIA Summit. This branch also assists in deploying the generator to http://droidgen.eventyay.com by triggering a travis build every time an apk is pushed to this branch. The reason this type of a round-about way was implemented is that, travis doesn’t allow android and docker builds on the same branch. So, we’re forced to use the apk branch to do the docker build for us.

Open Event Web application generator

A web application generator that allows event organisers to generate customised static easily-hostable web pages for their events.

Repository: fossasia/open-event-webapp.

The Open Event Web App project has two components :

  • An event website generator – A web application that generates the event website based on event data provided by the user either in the form of an archive as per the Open Event Universal Format or an API Endpoint to an Open Event API Server instance
  • A generic web application – This will be customised and additional pages will be generated from a set of templates by the generator based on the provided event data. The generated website bundle can be hosted on any static hosting provider (Github pages, any simple web hosting service, or any web server that can serve HTML files etc).

The generator also gives you the ability to directly upload the generated files to any server (via FTP) or to the gh-pages branch of any repository on GitHub.

  • Express.js – A web application framework for Node.js. The web application generator’s user-facing frontend runs on Express.js framework.
  • Socket.IO – A javascript library for real-time web applications. It allows a client and a server to communicate with each other simultaneously in a real-time manner. (Confusing ? If you wanna build something like a chat-room … you’d need something like Socket.IO). The web application generator uses Socket.IO to handle multiple clients and also to show the correct progress of the website generation to each of those clients in a real-time manner.
  • The web generator also uses SASS – which adds awesome features (Superpowers as their site says) to good-old CS, and Handlebars – which is a templating engine that let’s you easily generate HTML output based on a templates from a given javascript object.

The branches are similar to other projects under Open Event.

  • development – All development goes on in this branch. If you’re making a contribution, you are supposed to make a pull request to development. PRs to master must pass a build check and a unit-test (test/serverTest.js) check on Travis. Gets deployed automatically to https://opev-webgen-dev.herokuapp.com .
  • master – This contains shipped code. After significant features/bugfixes are accumulated on development, we make a version update, and make a release.

Open Event Data Scrappers

A scrapper that allows users to scrape events from popular websites such as eventbrite into the Open Event Universal Format for it to be imported into Open Event.

Repository: fossasia/event-collect.

As of now, only eventbrite is supported for data scrapping. This allows new users of Open Event to import their events from other ticketing/event management websites very easily. The scrapper accepts a query from the user, and gets the event page for that query on eventbrite, parses the HTML DOM to get the required data and compiles everything into the Open Event Universal Format so that it could be imported into Open Event.

The scrapper is written in Python and uses Beautiful Soup to parse the DOM.

Future plans include combining fossasia/query-server, Open Event server and event collect to enable automatic import of events from other websites based on user searches on eventyay.


The Open Event Ecosystem is vast and has plenty of contribution opportunities to both beginners and experts alike. Starting with contributions is as easy as picking a project, choosing an issue and submitting a PR. No strings attached. (Oh wait. there is just one string attached. Ensure you read & follow the “Best Practices at FOSSASIA Guide”)

{ Repost from my personal blog @ https://blog.codezero.xyz/open-event-ecosystem }

Continue ReadingThe Open Event Ecosystem

Generating xCal calendar in python

{ Repost from my personal blog @ https://blog.codezero.xyz/generate-xcal-calendar-in-python }

“xCal”, is an XML format for iCalendar data.

The iCalendar data format (RFC5545) is a widely deployed interchange format for calendaring and scheduling data.

A Sample xCal document

<?xml version="1.0" encoding="utf-8"?>  
<iCalendar xmlns:xCal="urn:ietf:params:xml:ns:xcal">  
    <vcalendar>
        <version>2.0</version>
        <prodid>-//Pentabarf//Schedule 1.0//EN</prodid>
        <x-wr-caldesc>FOSDEM 2016</x-wr-caldesc>
        <x-wr-calname>Schedule for events at FOSDEM 2016</x-wr-calname>
        <vevent>
            <method>PUBLISH</method>
            <uid>123e4567-e89b-12d3-a456-426655440000</uid>
            <dtstart>20160131T090000</dtstart>
            <dtend>20160131T091000</dtend>
            <duration>00:10:00:00</duration>
            <summary>Introduction to the SDR Track- Speakers, Topics, Algorithm</summary>
            <description>&lt;p&gt;The opening talk for the SDR devroom at FOSDEM 2016.&lt;/p&gt;</description>
            <class>PUBLIC</class>
            <status>CONFIRMED</status>
            <categories>Software Defined Radio</categories>
            <url>https:/fosdem.org/2016/schedule/event/sdrintro/</url>
            <location>AW1.125</location>
            <attendee>Martin Braun</attendee>
        </vevent>
    </vcalendar>
</iCalendar>

Each event/session will be in a seperate vevent block. Each speaker/attendee of an event/session will be in an attendee block inside a vevent block.

Some important elements are:

  1. version – Has the version of the iCalendar data
  2. prodid – Contains the name of the application/generator that generated this document
  3. x-wr-caldesc – A descriptive name for this calendar
  4. x-wr-calname – A description of the calendar

The structure and keywords used in xCal are the same as those used in the iCal format. To generate the XML document, we’ll be using python’s ElementTreeXML API that is part of the Python standard library.

We’ll be using two main classes of the ElementTree API:

  1. Element – used to create a standard node. (Used for the root node)
  2. SubElement – used to create a sub element and attache the new node to a parent

Let’s start with the root iCalendar node and set the required attributes.

from xml.etree.ElementTree import Element, SubElement, tostring

i_calendar_node = Element('iCalendar')  
i_calendar_node.set('xmlns:xCal', 'urn:ietf:params:xml:ns:xcal')

Now, to add the vcalendar node to the iCalendar node.

v_calendar_node = SubElement(i_calendar_node, 'vcalendar')

Let’s add the other aspects of the calendar to the vcalendar node as separate sub nodes.

version_node = SubElement(v_calendar_node, 'version')  
version_node.text = '2.0'

prod_id_node = SubElement(v_calendar_node, 'prodid')  
prod_id_node.text = '-//fossasia//open-event//EN'

cal_desc_node = SubElement(v_calendar_node, 'x-wr-caldesc')  
cal_desc_node.text = "Calendar"

cal_name_node = SubElement(v_calendar_node, 'x-wr-calname')  
cal_name_node.text = "Schedule for sessions"

Now, we have added information about our calendar. Now to add the actual events to the calendar. Each event would be a vevent node, a child of vcalendar node. We can loop through all our available event/sessions and add them to the calendar.

for session in sessions:  
    v_event_node = SubElement(v_calendar_node, 'vevent')

    uid_node = SubElement(v_event_node, 'uid')
    uid_node.text = str(session.id)

    dtstart_node = SubElement(v_event_node, 'dtstart')
    dtstart_node.text = session.start_time.isoformat()

    dtend_node = SubElement(v_event_node, 'dtend')
    dtend_node.text = tz.localize(session.end_time).isoformat()

    duration_node = SubElement(v_event_node, 'duration')
    duration_node.text =  "00:30"

    summary_node = SubElement(v_event_node, 'summary')
    summary_node.text = session.title

    description_node = SubElement(v_event_node, 'description')
    description_node.text = session.short_abstract

    class_node = SubElement(v_event_node, 'class')
    class_node.text = 'PUBLIC'

    status_node = SubElement(v_event_node, 'status')
    status_node.text = 'CONFIRMED'

    categories_node = SubElement(v_event_node, 'categories')
    categories_node.text = session.session_type.name

    url_node = SubElement(v_event_node, 'url')
    url_node.text = "https://some.conf/event/" + str(session.id)

    location_node = SubElement(v_event_node, 'location')
    location_node.text = session.microlocation.name

    for speaker in session.speakers:
        attendee_node = SubElement(v_event_node, 'attendee')
        attendee_node.text = speaker.name

Please note that all the timings in the XML Document must comply with ISO 8601 and must have the date+time+timezone. Example: 2007-04-05T12:30-02:00.

We’re still not done yet. We now have the XML document as an Element object. But we’ll be needing it as a string to either store it somewhere or display it.

The document can be converted to a string by using the ElementTree API’s tostring helper method and passing the root node.

xml_as_string = tostring(i_calendar_node)

And that’s it. You now have a proper XML document representing your events.

Continue ReadingGenerating xCal calendar in python

Building the Scheduler UI

{ Repost from my personal blog @ https://blog.codezero.xyz/building-the-scheduler-ui }

If you hadn’t already noticed, Open Event has got a shiny new feature. A graphical and an Interactive scheduler to organize sessions into their respective rooms and timings.

As you can see in the above screenshot, we have a timeline on the left. And a lot of session boxes to it’s right. All the boxes are re-sizable and drag-drop-able. The columns represent the different rooms (a.k.a micro-locations). The sessions can be dropped into their respective rooms. Above the timeline, is a toolbar that controls the date. The timeline can be changed for each date by clicking on the respective date button.

The Clear overlaps button would automatically check the timeline and remove any sessions that are overlapping each other. The Removed sessions will be moved to the unscheduled sessions pane at the left.

The Add new micro-location button can be used to instantly add a new room. A modal dialog would open and the micro-location will be instantly added to the timeline once saved.

The Export as iCal allows the organizer to export all the sessions of that event in the popular iCalendar format which can then be imported into various calendar applications.

The Export as PNG saves the entire timeline as a PNG image file. Which can then be printed by the organizers or circulated via other means if necessary.

Core Dependencies

The scheduler makes use of some javascript libraries for the implementation of most of the core functionality

  • Interact.js – For drag-and-drop and resizing
  • Lodash – For array/object manipulations and object cloning
  • jQuery – For DOM Manipulation
  • Moment.js – For date time parsing and calculation
  • Swagger JS – For communicating with our API that is documented according to the swagger specs.
Retrieving data via the API

The swagger js client is used to obtain the sessions data using the API. The client is asynchronously initialized on page load. The client can be accessed from anywhere using the javascript function initializeSwaggerClient.

The swagger initialization function accepts a callback which is called if the client is initialized. If the client is not initialized, the callback is called after that.

var swaggerConfigUrl = window.location.protocol + "//" + window.location.host + "/api/v2/swagger.json";  
window.swagger_loaded = false;  
function initializeSwaggerClient(callback) {  
    if (!window.swagger_loaded) {
        window.api = new SwaggerClient({
            url: swaggerConfigUrl,
            success: function () {
                window.swagger_loaded = true;
                if (callback) {
                    callback();
                }

            }
        });
    } else {
        if (callback) {
            callback();
        }
    }
}

For getting all the sessions of an event, we can do,

initializeSwaggerClient(function () {  
    api.sessions.get_session_list({event_id: eventId}, function (sessionData) {
        var sessions = sessionData.obj;  // Here we have an array of session objects
    });
});

In a similar fashion, all the micro-locations of an event can also be loaded.

Processing the sessions and micro-locations

Each session object is looped through, it’s start time and end time are parsed into moment objects, duration is calculated, and it’s distance from the top in the timeline is calculated in pixels. The new object with additional information, is stored in an in-memory data store, with the date of the event as key, for use in the timeline.

The time configuration is specified in a separate time object.

var time = {  
    start: {
        hours: 0,
        minutes: 0
    },
    end: {
        hours: 23,
        minutes: 59
    },
    unit: {
        minutes: 15,
        pixels: 48,
        count: 0
    },
    format: "YYYY-MM-DD HH:mm:ss"
};

The smallest unit of measurement is 15 minutes and 48px === 15 minutes in the timeline.

Each day of the event is stored in a separate array in the form of Do MMMM YYYY(eg. 2nd May 2013).

The array of micro-location objects is sorted alphabetically by the room name.

Displaying sessions and micro-locations on the timeline

According to the day selected, the sessions for that day are displayed on the timeline. Based on their time, the distance of the session div from the top of the timeline is calculated in pixels and the session box is positioned absolutely. The height of the session in pixels is calculated from it’s duration and set.

For pixels-minutes conversion, the following are used.

/**
 * Convert minutes to pixels based on the time unit configuration
 * @param {number} minutes The minutes that need to be converted to pixels
 * @returns {number} The pixels
 */
function minutesToPixels(minutes) {  
    minutes = Math.abs(minutes);
    return (minutes / time.unit.minutes) * time.unit.pixels;
}

/**
 * Convert pixels to minutes based on the time unit configuration
 * @param {number} pixels The pixels that need to be converted to minutes
 * @returns {number} The minutes
 */
function pixelsToMinutes(pixels) {  
    pixels = Math.abs(pixels);
    return (pixels / time.unit.pixels) * time.unit.minutes;
}
Adding interactivity to the session elements

Interact.js is used to provide interactive capabilities such as drag-drop and resizing.

To know how to use Interact.js, you can checkout some previous blog posts on the same, Interact.js + drag-drop and Interact.js + resizing.

Updating the session information in database on every change

We have to update the session information in database whenever it is moved or resized. Every time a session is moved or resized, a jQuery event is triggered on $(document) along with the session object as the payload.

We listen to this event, and make an API request with the new session object to update the session information in the database.


The scheduler UI is more complex than said in this blog post. To know more about it, you can checkout the scheduler’s javascript code atapp/static/js/admin/event/scheduler.js.

Continue ReadingBuilding the Scheduler UI

Accepting Stripe payments on behalf of a third-party

{ Repost from my personal blog @ https://blog.codezero.xyz/accepting-stripe-payments-on-behalf-of-a-third-party }

In Open Event, we allow the organizer of each event to link their Stripe account, so that all ticket payments go directly into their account. To make it simpler for the organizer to setup the link, we have a Connect with stripe button on the event creation form.

Clicking on the button, the organizer is greeted with a signup flow similar to Login with Facebook or any other social login. Through this process, we’re able to securely and easily obtain the credentials required to accept payments on behalf of the organizer.

For this very purpose, stripe provides us with an OAuth interface called as Stripe Connect. Stripe Connect allows us to connect and interact with other stripe accounts through an API.

We’ll be using Python’s requests library for making all the HTTP Requests to the API.
You will be needing a stripe account for this.

Registering your platform
The OAuth Flow

The OAuth flow is similar to most platforms.

  • The user is redirected to an authorization page where they login to their stripe account and authorize your app to access their account
  • The user is then redirected back to a callback URL with an Authorization code
  • The server makes a request to the Token API with the Authorization code to retrieve the access_token, refresh_token and other credentials.

Implementing the flow

Redirect the user to the Authorization URL.
https://connect.stripe.com/oauth/authorize?response_type=code&client_id=ca_8x1ebxrl8eOwOSqRTVLUJkWtcfP92YJE&scope=read_write&redirect_uri=http://localhost/stripe/callback  

The authorization url accepts the following parameters.

  1. client_id – The client ID acquired when registering your platform.required.
  2. response_type – Response type. The value is always code. required.
  3. redirect_uri – The URL to redirect the customer to after authorization.
  4. scope – Can be read_write or read_only. The default is read_only. For analytics purposes, read_only is appropriate; To perform charges on behalf of the connected user, We will need to request read_write scope instead.

The user will be taken to stripe authorization page, where the user can login to an existing account or create a new account without breaking the flow. Once the user has authorized the application, he/she is taken back to the Callback URL with the result.

Requesting the access token with the authorization code

The user is redirected back to the callback URL.

If the authorization failed, the callback URL has a query string parameter error with the error name and a parameter error_description with the description of the error.

If the authorization was a success, the callback URL has the authorization code in the code query string parameter.

import requests

data = {  
    'client_secret': 'CLIENT_SECRET',
    'code': 'AUTHORIZATION_CODE',
    'grant_type': 'authorization_code'
}

response = requests.post('https://connect.stripe.com/oauth/token', data=data)

The client_secret is also obtained when registering your platform. The codeparameter is the authorization code.

On making this request, a json response will be returned.

If the request was a success, the following response will be obtained.

{
  "token_type": "bearer",
  "stripe_publishable_key": PUBLISHABLE_KEY,
  "scope": "read_write",
  "livemode": false,
  "stripe_user_id": USER_ID,
  "refresh_token": REFRESH_TOKEN,
  "access_token": ACCESS_TOKEN
}

If the request failed for some reason, an error will be returned.

{
  "error": "invalid_grant",
  "error_description": "Authorization code does not exist: AUTHORIZATION_CODE"
}

The access_token token obtained can be used as the secret key to accept payments like discussed in Integrating Stripe in the Flask web framework.

Continue ReadingAccepting Stripe payments on behalf of a third-party

PayPal Express Checkout in Python

As per the PayPal documentation …

Express Checkout is a fast, easy way for buyers to pay with PayPal. Express Checkout eliminates one of the major causes of checkout abandonment by giving buyers all the transaction details at once, including order details, shipping options, insurance choices, and tax totals.

The basic steps for using express checkout to receive one-time payments are:

  1. Getting the PayPal API credentials.
  2. Making a request to the API with the transaction details to get a token
  3. Using the token to send the users to the PayPal payment page
  4. Capturing the payment and charging the user after the user completes the payment at PayPal.

We will be using PayPal’s Classic NVP (Name-value pair) API for implementing this.

Getting PayPal API Credentials

To begin with, we’ll need API Credentials.
We’ll be using the Signature API credentials which consists of

  • API Username
  • API Password
  • Signature

To obtain these, you can follow the steps at Creating and managing NVP/SOAP API credentials – PayPal Developer.

You’ll be getting two sets of credentials. Sandbox and Live. We’ll just stick to the Sandbox for now.

Now, we need sandbox test accounts for making and receiving payments. Head over to Creating Sandbox Test Accounts – PayPal Developer and create two sandbox test accounts. One would be the facilitator and one would be the buyer.

PayPal NVP Servers

All the API actions will take place by making a request to the PayPal server. PayPal has 4 different NVP servers for 4 different purposes.

  1. https://api-3t.sandbox.paypal.com/nvp – Sandbox “testing” server for use with API signature credentials.
  2. https://api-3t.paypal.com/nvp– PayPal “live” production server for use with API signature credentials.
  3. https://api.sandbox.paypal.com/nvp – Sandbox “testing” server for use with API certificate credentials.
  4. https://api.paypal.com/nvp – PayPal “live” production server for use with API certificate credentials.

We’ll be using the Sandbox “testing” server for use with API signature credentials.

Creating a transaction and obtaining the token

To create a transaction, we’ll need to make a request with all the transaction details. We can use Python requests library to easily make the requests. All requests are POST.

We’ll be calling the SetExpressCheckout method of the NVP API to obtain the token.

import requests  
import urlparse

data = {  
    'USER': credentials['USER'],
    'PWD': credentials['PWD'],
    'SIGNATURE': credentials['SIGNATURE'],
    'SUBJECT': credentials['FACILITATOR_EMAIL'],
    'METHOD': 'SetExpressCheckout',
    'VERSION': 93,
    'PAYMENTREQUEST_0_PAYMENTACTION': 'SALE',
    'PAYMENTREQUEST_0_AMT': 100,
    'PAYMENTREQUEST_0_CURRENCYCODE': 'USD',
    'RETURNURL': 'http://localhost:5000/paypal/return/',
    'CANCELURL': 'http://localhost:5000/paypal/cancel/'
}
response = requests.post('https://api-3t.sandbox.paypal.com/nvp', data=data)  
token = dict(urlparse.parse_qsl(response.text))['TOKEN']

Here,

  • USER represents your Sandbox API Username.
  • PWD represents your Sanbox API Password.
  • SIGNATURE represents your Sandbox Signature.
  • SUBJECT represents the facilitator’s email ID.
  • PAYMENTREQUEST_0_AMT is the total transaction amount.
  • PAYMENTREQUEST_0_CURRENCYCODE is the 3 digit ISO 4217 Currency code.
  • RETURNURL is where the user will be sent to after the transaction
  • CANCELURL is where the user will be sent to if he/she cancels the transaction.

A URL-Encoded, Name-value pair response would be obtained. We can decode that into a dict by using Python’s urlparse modules.

From the response, we’re extracting the TOKEN which we will use to generate the payment URL for the user.

This token has to be retained since we’ll be using it in further steps of the process.

Redirecting the user to PayPal for Approval

With the token we obtained, we can form the payment URL.

https://www.sandbox.paypal.com/cgi-bin/webscr?cmd=_express-checkout&token=<TOKEN>

We’ll have to send the user to that URL. Once the user completes the transaction at PayPal, he/she will be returned to the RETURNURL where we’ll further process the transaction.

Obtaining approved payment details and capturing the payment

Once the user completes the transaction and gets redirected back to RETURNURL, we’ll have to obtain the confirmed payment details from PayPal. For that we can again use the token ID that we obtained before.

We’ll now be making a request to the GetExpressCheckoutDetails method of the API.

import requests  
import urlparse

data = {  
    'USER': credentials['USER'],
    'PWD': credentials['PWD'],
    'SIGNATURE': credentials['SIGNATURE'],
    'SUBJECT': credentials['FACILITATOR_EMAIL'],
    'METHOD': 'GetExpressCheckoutDetails',
    'VERSION': 93,
    'TOKEN': TOKEN
}

response = requests.post('https://api-3t.sandbox.paypal.com/nvp', data=data)  
result = dict(urlparse.parse_qsl(response.text))  
payerID = result['PAYERID']

A URL-Encoded, Name-value pair response would be obtained. We can decode that into a dict by using Python’s urlparse modules.

This will provide us with information about the transaction such as transaction time, transaction amount, charges, transaction mode, etc.

But, we’re more interested in the PAYERID which we’ll need to capture/collect the payment. The money is not transferred to the facilitators account until it is captured/collected. So, be sure to collect it.

To collect it, we’ll be making another request to the DoExpressCheckoutPaymentmethod of the API using the token and the PAYERID.

import requests  
import urlparse

data = {  
    'USER': credentials['USER'],
    'PWD': credentials['PWD'],
    'SIGNATURE': credentials['SIGNATURE'],
    'SUBJECT': credentials['FACILITATOR_EMAIL'],
    'METHOD': 'DoExpressCheckoutPayment',
    'VERSION': 93,
    'TOKEN': TOKEN,
    'PAYERID': payerID,
    'PAYMENTREQUEST_0_PAYMENTACTION': 'SALE',
    'PAYMENTREQUEST_0_AMT': 100,
    'PAYMENTREQUEST_0_CURRENCYCODE': 'USD',
}

response = requests.post('https://api-3t.sandbox.paypal.com/nvp', data=data)  
result = dict(urlparse.parse_qsl(response.text))  
status = result['ACK']

All the details have to be the same as the ones provided while obtaining the token. Once we make the request, we’ll again get a URL-Encoded, Name-value pair response. We can decode that into a dict by using Python’s urlparsemodules.

From the response, ACK (Acknowledgement status) will provide us with the status of the payment.

  • Success — A successful operation.
  • SuccessWithWarning — A successful operation; however, there are messages returned in the response that you should examine.
  • Failure — The operation failed; the response also contains one or more error messages explaining the failure.
  • FailureWithWarning — The operation failed and there are messages returned in the response that you should examine.

And, we have completed the PayPal transaction flow for Express Checkout. These are just the basics and might miss a few stuff. I suggest you go through the following links too for a better understanding of everything:

For Reference:
  1. PayPal Name-Value Pair API Basics – PayPal Developer
  2. How to Create One-Time Payments Using Express Checkout – PayPal Developer
Continue ReadingPayPal Express Checkout in Python

Integrating Stripe in the Flask web framework

{ Repost from my personal blog @ https://blog.codezero.xyz/integrating-stripe-in-flask }

Stripe is a developer and a user-friendly payment infrastructure provider. Stripe provides easy to use SDKs in different programming languages allowing us to easily collect payments on our website or mobile application.

Flask is a web microframework for Python based on Werkzeug, Jinja 2. Flask makes building web applications in python a breeze.

Make sure you have your Flask app ready. Let’s start with installing the required dependency. The Stripe python SDK. You can get it by running.

pip install stripe

Don’t forget to add the same in your requirements.txt. (if you have one that is.)

Now, head over to Stripe: Register and create a new Stripe account to get your test keys. If you don’t wish to create an account at this time, you can use the following test keys, but you’ll not be able to see the payments in the stripe dashboard.

  • Publishable Key: pk_test_6pRNASCoBOKtIshFeQd4XMUh
  • Secret Key: sk_test_BQokikJOvBiI2HlWgH4olfQ2

We’ll need to set the secret key in the SDK.

import stripe

STRIPE_PUBLISHABLE_KEY = 'pk_test_6pRNASCoBOKtIshFeQd4XMUh'  
STRIPE_SECRET_KEY = 'sk_test_BQokikJOvBiI2HlWgH4olfQ2'

stripe.api_key = STRIPE_SECRET_KEY

Let’s create a page with a form for us to handle the Stripe payment.

<!DOCTYPE html>  
<html>  
<head>  
    <title>Pay now</title>
</head>  
<body>  
    <h4>Pay $250.00 by clicking on the button below.</h4>
    <form action="/payment" method="POST">
        <script src="https://checkout.stripe.com/checkout.js" 
                class="stripe-button"
                data-key="pk_test_6pRNASCoBOKtIshFeQd4XMUh"
                data-description="A payment for the Hello World project"
                data-name="HelloWorld.com"
                data-image="/images/logo/hw_project.png"
                data-amount="25000"></script>
    </form>
</body>  
</html>

We’re using Stripe’s Checkout library to get the payment details from the user and process. Also, keep in mind that the checkout library has to be loaded directly from https://checkout.stripe.com/checkout.js. Downloading it and serving locally will not work.

The script tag, accepts a lot of parameters. A few important ones are,

  • data-key – The Publishable Key.
  • data-amount – The amount to be charged to the user in the lowest denomination of the currency. (For example, 5 USD should be represented as 500 cents)
  • data-name – The name of your site or company that will be displayed to the user.
  • data-image – The path to an image file (maybe a logo) that you’d like to be displayed to the user.

More configuration options can be seen at Stripe: Detailed Checkout Guide.

This script would automatically create a Pay with Card button which would open the stripe Checkout lightbox when clicked by the user.

Once the payment process is completed the following parameters are submitted to the form’s action endpoint (the form inside which this script is located), along with any other elements that were in the form.

  • stripeToken – The ID of the token representing the payment details
  • stripeEmail – The email address the user entered during the Checkout process

Along with the Billing address details and Shipping address details if applicable and enabled

We’ll need to write a Flask method to handle the input that were submitted by Stripe to proceed with the transaction and charge the user.

Let’s add a new Flask route to respond when submitting the form.

@app.route('/payment', methods=['POST'])
def payment_proceed():  
    # Amount in cents
    amount = 25000

    customer = stripe.Customer.create(
        email=request.form['stripeEmail'],
        source=request.form['stripeToken']
    )

    charge = stripe.Charge.create(
        amount=amount,
        currency='usd',
        customer=customer.id,
        description='A payment for the Hello World project'
    )

    return render_template('payment_complete.html')

We’re now creating a new Stripe customer along with the stripeToken as the source parameter. The card details are stored by stripe as a token. And using this token ID, Stripe will be able to retrieve it to make the charge.

We’re creating a charge object with the amount in the lowest denomination of the currency, the currency name, the customer ID, and an optional description. This will charge the customer. On a successful transaction, a charge object would be returned. Else, an exception will be thrown.

For more information regarding the Charge object and the various other APIs available fro consumption in Stripe, checkout the Stripe API Guide.

Continue ReadingIntegrating Stripe in the Flask web framework

Building a logger interface for FlightGear using Python: Part One

{ Repost from my personal blog @ https://blog.codezero.xyz/python-logger-interface-for-flightgear-part-one/ }

The FlightGear flight simulator is an open-source, multi-platform, cooperative flight simulator developed as a part of the FlightGear project. I have been using this Flight simulator for a year for Virtual Flight testing, running simulations and measuring flight parameters during various types of maneuvers. I have noticed that, logging the data, (figuring out how to log in the first place) has been quite difficult for users with less technical knowledge in such softwares.

Also, the Property Tree of FlightGear is pretty extensive making it difficult to properly traverse the huge tree to get the parameters that are actually required.

That’s when I got the idea of making a simple, easy to use, user friendly logging interface for FlightGear. I gave it a name ‘FlightGear Command Center’:wink: and the project was born at github.com/niranjan94/flightgear-cc.

After 44 commits, this is what I have now.

1. A simple dashboard to connect to FlightGear, open FlightGear with a default plane, Getting individual parameter values or to log a lot of parameters continuously

2. An interface to choose the parameters to log and the interval

  1. The User interface is a web application written in HTML/javascript.
  2. The Web application communicates with a python bridge using WebSockets.
  3. The python bridge communicates with FlightGear via telnet.
  4. The data is logged to a csv file continuously (until the user presses stop) by the bridge once the web application requests it.
The interface with FlightGear

FlightGear has an internal “telnet” command server which provides us “remote shell” into the running FlightGear process which we can exploit to interactively view or modify any property/variable of the simulation.

FlightGear can be instructed to start the server and listen for commands by passing the --telnet=socket,out,60,localhost,5555,udp command line argument while starting FlightGear. (The argument is of format --telnet=medium,direction,speed_in_hertz,localhost,PORT,style.)

Communication with that server can be done using any simple telnet interface. But FlightGear also provides us with a small wrapper class that makes retrieving and setting properties using the telnet server even more easier.

The wrapper can be obtained from the official repository atsourceforge.net/p/flightgear/flightgear/ci/master/tree/scripts/python/FlightGear.py

Using the wrapper is straightforward. Initialize an instance of the class with the hostname and port. The class will then make a connection to the telnet server.

from FlightGear import FlightGear

flightgear_server = 'localhost'  
flightgear_server_port = 5555  
fg = FlightGear(flightgear_server, flightgear_server_port)

The wrapper makes use of python’s magic methods __setitem__ and __getitem__ to make it easy for us to read or manipulate the property tree.

For example, getting the current altitude of the airplane is as easy as

print fg['/position[0]/altitude-ft']

and setting the altitude is as simple as

fg['/position[0]/altitude-ft'] = 345.2

But the important thing here is, knowing the path to the data you want in the FlightGear property tree. Most of the commonly used properties are available over at Aircraft properties reference – FlightGear Wiki.

Now that we have basic interface between python and FlightGear in place, the next step would be to setup a link between the user interface (a small web app) and the python bridge. We would be using WebSockets for that so as to have a Real-time and an always on link to the bridge which would enable us to in turn communicate with FlightGear in realtime.

We need a WebSocket server in place. So, I used the SimpleWebSocketServer.pyclass from github.com/dpallot/simple-websocket-server.

A websocket server can be created by,

from SimpleWebSocketServer import SimpleWebSocketServer, WebSocket

hostname = 'localhost'  
websocket_server_port = 8888

class SocketHandler(WebSocket):

    def handleMessage(self):
        # print the message when received 
        print self.data

    def handleConnected(self):
        print self.address, 'connected'

    def handleClose(self):
        print self.address, 'closed'

server = SimpleWebSocketServer(hostname, websocket_server_port, SocketHandler)  
server.serveforever()
  • handleMessage is called whenever a client sends a message to the server
  • handleConnected is called when a new client connects to the server
  • handleClose is called when a client disconnects from the server

A message can be sent to the clients by using the sendMessage method from within the SocketHandler.

class SocketHandler(WebSocket):

    def handleMessage(self):
        # send a hello whenever a message is received  
        print self.data
        self.sendMessage('Hello')

    def handleConnected(self):
        print self.address, 'connected'

    def handleClose(self):
        print self.address, 'closed'

We now have a WebSocket server in place. Now the web app can easily talk to this server using javascript websockets API. Which would be continued in upcoming blog articles.

Continue ReadingBuilding a logger interface for FlightGear using Python: Part One

Building interactive elements with HTML and javascript: Interact.js + resizing

{ Repost from my personal blog @ https://blog.codezero.xyz/building-interactive-elements-with-html-and-javascript-interact-js-resizing/ }

In a few of the past blog posts, we saw about implementing resizing with HTML and javascript. The functionality was pretty basic with simple resizing. In the last blog post we saw about interact.js.

interact.js is a lightweight, standalone JavaScript module for handling single-pointer and multi-touch drags and gestures with powerful features including inertia and snapping.

Getting started with Interact.js

You have multiple option to include the library in your project.

  • You can use bower to install (bower install interact) (or)
  • npm (npm install interact.js) (or)
  • You could directly include the library from a CDN (https://cdnjs.cloudflare.com/ajax/libs/interact.js/1.2.6/interact.min.js).
Implementing resizing

Let’s create a simple box using HTML. We’ll add a class called resizable to it so that we can reference it to initialize Interact.js

<div class="resizable">  
    Use right/bottom edge to resize
</div>

We need to create an interact instance. Once the instance is created, we have to call the resizable method on it to add resize support to the div.

interact('.resizable')
  .resizable({
    edges: { right: true, bottom: true }
  })
  .on('resizemove', function (event) {
    

  });

Inside the resizable method, we can pass configuration options. The edgesconfig key allows us to specify on which all edges, resizing should be allowed. Right now, we have allowed on the right and bottom edges. Similarly we can have resizing support in the top and left edges too.

The resizemove event is triggered by interact every time the user tries to resize the div. From the event, we can get the box that is being resized, (i.e) the target by accessing event.target.

The event object also provides us event.rect.width and event.rect.height which is the width and height of the div after resizing. We’ll not set this as the width of the div so that, the user is able to see the width change.

var target = event.target;
    // update the element's style
    target.style.width  = event.rect.width + 'px';
    target.style.height = event.rect.height + 'px';

We can also instruct Interact.js to preserve the aspect ratio of the box by adding an option preserveAspectRatio: true to the configuration object passed to resizable method during initialization.

JavaScript
interact('.resizable')
  .resizable({
    edges: { right: true, bottom: true }
  })
  .on('resizemove', function (event) {
    var target = event.target;

    // update the element's style
    target.style.width  = event.rect.width + 'px';
    target.style.height = event.rect.height + 'px';
  });

Resizing and drag-drop (with Interact.js) were used to create the Scheduler tool at Open Event. The tool allows event/track organizers to easily arrange the sessions into their respective rooms by drag-drop and also to easily change the timings of the events by resizing the event block. The entire source code of the scheduler can be viewed at app/static/js/admin/event/scheduler.js in the Open Event Organizer server’s GitHub repository.

Demo:
https://jsfiddle.net/xdfocdty/

Continue ReadingBuilding interactive elements with HTML and javascript: Interact.js + resizing

Building interactive elements with HTML and javascript: Interact.js + drag-drop

{ Repost from my personal blog @ https://blog.codezero.xyz/building-interactive-elements-with-html-and-javascript-interact-js-drag-drop }

In a few of the past blog posts, we saw about implementing drag-drop andresizing with HTML and javascript. The functionality was pretty basic with a simple drag-and-drop and resizing. That is where, a javascript library called as interact.js comes in.

interact.js is a lightweight, standalone JavaScript module for handling single-pointer and multi-touch drags and gestures with powerful features including inertia and snapping.

With Interact.js, building interactive elements is like a eating a piece of your favorite cake – that easy !

Getting started with Interact.js

You have multiple option to include the library in your project.

  • You can use bower to install (bower install interact) (or)
  • npm (npm install interact.js) (or)
  • You could directly include the library from a CDN (https://cdnjs.cloudflare.com/ajax/libs/interact.js/1.2.6/interact.min.js).

Implementing a simple draggable

Let’s start with some basic markup. We’ll be using the draggable class to enable interact.js on this element.

<div id="box-one" class="draggable">  
  <p> I am the first Box </p>
</div>  
<div id="box-two" class="draggable">  
    <p> I am the second Box </p>
</div>

The first step in using interact.js is to create an interact instance. Which you can create by using interact('<the selector>'). Once the instance is created, you’ll have to call the draggable method on it to enable drag. Draggable accepts a javascript object with some configuration options and some pretty useful callbacks.

// target elements with the "draggable" class
interact('.draggable')  
  .draggable({
    // enable inertial throwing
    inertia: true,
    // keep the element within the area of it's parent
    restrict: {
      restriction: "parent",
      endOnly: true,
      elementRect: { top: 0, left: 0, bottom: 1, right: 1 }
    },
    // enable autoScroll
    autoScroll: true,
    // call this function on every dragmove event
    onmove: dragMoveListener,
  });

  function dragMoveListener (event) {
    var target = event.target,
        // keep the dragged position in the data-x/data-y attributes
        x = (parseFloat(target.getAttribute('data-x')) || 0) + event.dx,
        y = (parseFloat(target.getAttribute('data-y')) || 0) + event.dy;

    // translate the element
    target.style.webkitTransform =
    target.style.transform =
      'translate(' + x + 'px, ' + y + 'px)';

    // update the posiion attributes
    target.setAttribute('data-x', x);
    target.setAttribute('data-y', y);
  }

Here we use the onmove event to move the box according to the dx and dyprovided by interact when the element is dragged.

Implementing a simple drag-drop

Now to the above draggable, we’ll add a drop zone into which the two draggable boxes can be dropped.

<div id="dropzone" class="dropzone">You can drop the boxes here</div>

Similar to a draggable, we first create an interact instance. Then we call the dropzone method on to tell interact that, that div is to be considered as a dropzone. The dropzone method accepts a json object with configuration options and callbacks.

// enable draggables to be dropped into this
interact('.dropzone').dropzone({  
  // Require a 50% element overlap for a drop to be possible
  overlap: 0.50,

  // listen for drop related events:

  ondropactivate: function (event) {
    // add active dropzone feedback
    event.target.classList.add('drop-active');
  },
  ondragenter: function (event) {
    var draggableElement = event.relatedTarget,
        dropzoneElement = event.target;

    // feedback the possibility of a drop
    dropzoneElement.classList.add('drop-target');
  },
  ondragleave: function (event) {
    // remove the drop feedback style
    event.target.classList.remove('drop-target');
  },
  ondrop: function (event) {
    event.relatedTarget.textContent = 'Dropped';
  },
  ondropdeactivate: function (event) {
    // remove active dropzone feedback
    event.target.classList.remove('drop-active');
    event.target.classList.remove('drop-target');
  }
});

Each event provides two important properties. relatedTarget which gives us the DOM object of the draggable that is being dragged. And target which gives us the DOM object of the dropzone. We can use this to provide visual feedback for the user when he/she is dragging.

Demo:

https://jsfiddle.net/niranjan94/yqwc4hqz/2/

Continue ReadingBuilding interactive elements with HTML and javascript: Interact.js + drag-drop

Implementing revisioning feature in Open Event

{ Repost from my personal blog @ https://blog.codezero.xyz/implementing-revisioning-feature-in-open-event }

As I said in my previous blog post about Adding revisioning to SQLAlchemy Models,

In an application like Open Event, where a single piece of information can be edited by multiple users, it’s always good to know who changed what. One should also be able to revert to a previous version if needed.

Let’s have a quick run through on how we can enable SQLAlchemy-Continuum on our project.

  1. Install the library SQLAlchemy-Continuum with pip
  2. Add __versioned__ = {} to all the models that need to be versioned.
  3. Call make_versioned() before the models are defined
  4. Call configure_mappers from SQLAlchemy after declaring all the models.

Example:

import sqlalchemy as sa  
from sqlalchemy_continuum import make_versioned

# Must be called before defining all the models
make_versioned()

class Event(Base):

    __tablename__ = 'events'
    __versioned__ = {}  # Must be added to all models that are to be versioned

    id = sa.Column(sa.Integer, primary_key=True, autoincrement=True)
    name = sa.Column(sa.String)
    start_time = sa.Column(db.DateTime, nullable=False)
    end_time = sa.Column(db.DateTime, nullable=False)
    description = db.Column(db.Text)
    schedule_published_on = db.Column(db.DateTime)

# Must be called after defining all the models
sa.orm.configure_mappers()

We have SQLAlchemy-Continuum enabled now. You can do all the read/write operations as usual. (No change there).

Now, for the part where we give the users an option to view/restore revisions. The inspiration for this, comes from wordpress’s wonderful revisioning functionality.

The layout is well designed. The differences are shown in an easy-to-read form. The slider on top makes it intuitive to move b/w revisions. We have a Restore this Revision button on the top-right to switch to that revision.

A similar layout is what we would like to achieve in Open Event.

  1. A slider to switch b/w sessions
  2. A pop-over infobox on the slider to show who made that change
  3. A button to switch to that selected revision.
  4. The colored-differences shown in side-by-side manner.

To make all this a bit easier, SQLAlchemy-Continuum provides us with some nifty methods.

count_versions is a method that allows us to know the number of revisions a record has.

event = session.query(Event).get(1)  
count = count_versions(event)  # number of versions of that event

Next one is pretty cool. All the version objects have a property called as changeset which holds a dict of changed fields in that version.

event = Event(name=u'FOSSASIA 2016', description=u'FOSS Conference in Asia')  
session.add(article)  
session.commit(article)

version = event.versions[0]  # first version  
version.changeset  
# {
#   'id': [None, 1],
#   'name': [None, u'FOSSASIA 2016'],
#   'description': [None, u'FOSS Conference in Asia']
# }

event.name = u'FOSSASIA 2017'  
session.commit()

version = article.versions[1]  # second version  
version.changeset  
# {
#   'name': [u'FOSSASIA 2016', u'FOSSASIA 2017'],
# }

As you can see, dict holds the fields that changed and the content the changed (before and after). And this is what we’ll be using for generating those pretty diffs that the guys and girls over at wordpress.com have done. And for this we’ll be using two things.

  1. A library named diff-match-patch. It is a library from Google which offers robust algorithms to perform the operations required for synchronizing plain text.
  2. A small recipe from from code.activestate.com Line-based diffs with the necessary HTML markup for styling insertions and deletions.
import itertools  
import re

import diff_match_patch

def side_by_side_diff(old_text, new_text):  
    """
    Calculates a side-by-side line-based difference view.

    Wraps insertions in <ins></ins> and deletions in <del></del>.
    """
    def yield_open_entry(open_entry):
        """ Yield all open changes. """
        ls, rs = open_entry
        # Get unchanged parts onto the right line
        if ls[0] == rs[0]:
            yield (False, ls[0], rs[0])
            for l, r in itertools.izip_longest(ls[1:], rs[1:]):
                yield (True, l, r)
        elif ls[-1] == rs[-1]:
            for l, r in itertools.izip_longest(ls[:-1], rs[:-1]):
                yield (l != r, l, r)
            yield (False, ls[-1], rs[-1])
        else:
            for l, r in itertools.izip_longest(ls, rs):
                yield (True, l, r)

    line_split = re.compile(r'(?:r?n)')
    dmp = diff_match_patch.diff_match_patch()

    diff = dmp.diff_main(old_text, new_text)
    dmp.diff_cleanupSemantic(diff)

    open_entry = ([None], [None])
    for change_type, entry in diff:
        assert change_type in [-1, 0, 1]

        entry = (entry.replace('&', '&amp;')
                      .replace('<', '&lt;')
                      .replace('>', '&gt;'))

        lines = line_split.split(entry)

        # Merge with previous entry if still open
        ls, rs = open_entry

        line = lines[0]
        if line:
            if change_type == 0:
                ls[-1] = ls[-1] or ''
                rs[-1] = rs[-1] or ''
                ls[-1] = ls[-1] + line
                rs[-1] = rs[-1] + line
            elif change_type == 1:
                rs[-1] = rs[-1] or ''
                rs[-1] += '<ins>%s</ins>' % line if line else ''
            elif change_type == -1:
                ls[-1] = ls[-1] or ''
                ls[-1] += '<del>%s</del>' % line if line else ''

        lines = lines[1:]

        if lines:
            if change_type == 0:
                # Push out open entry
                for entry in yield_open_entry(open_entry):
                    yield entry

                # Directly push out lines until last
                for line in lines[:-1]:
                    yield (False, line, line)

                # Keep last line open
                open_entry = ([lines[-1]], [lines[-1]])
            elif change_type == 1:
                ls, rs = open_entry

                for line in lines:
                    rs.append('<ins>%s</ins>' % line if line else '')

                open_entry = (ls, rs)
            elif change_type == -1:
                ls, rs = open_entry

                for line in lines:
                    ls.append('<del>%s</del>' % line if line else '')

                open_entry = (ls, rs)

    # Push out open entry
    for entry in yield_open_entry(open_entry):
        yield entry

So, what we have to do is,

  1. Get the changeset from a version
  2. Run each field’s array containing the old and new text through the side_by_side_diff method.
  3. Display the output on screen.
  4. Use the markups <ins/> and <del/> to style changes.

So, we do the same for each version by looping through the versions array accessible from an event record.

For the slider, noUiSlider javascript library was used. Implementation is simple.

<div id="slider"></div>

<script type="text/javascript">  
    $(function () {
        var slider = document.getElementById('slider');
        noUiSlider.create(slider, {
            start: [0],
            step: 1,
            range: {
                'min': 0,
                'max': 5
            }
        });
    });
</script>

This would create a slider that can go from 0 to 5 and will start at position 0.

By listening to the update event of the slider, we’re able to change which revision is displayed.

slider.noUiSlider.on('update', function (values, handle) {  
    var value = Math.round(values[handle]);
    // the current position of the slider
    // do what you have to do to change the displayed revision
});

And to get the user who caused a revision, you have to access the user_idparameter of the transaction record of a particular version.

event = session.query(Event).get(1)  
version_one = event.versions[0]  
transaction = transaction_class(version_one)  
user_id = transaction.user_id

So, with the user ID, you can query the user database to get the user who made that revision.

The user_id is automatically populated if you’re using Flask, Flask-login and SQLAlchemy-Continuum’s Flask Plugin. Enabling the plugin is easy.

from sqlalchemy_continuum.plugins import FlaskPlugin  
from sqlalchemy_continuum import make_versioned

make_versioned(plugins=[FlaskPlugin()])

This is not a very detailed blog post. If you would like to see the actual implementation, you can checkout the Open Event repository over at GitHub. Specifically, the file browse_revisions.html.

The result is,

Still needs some refinements in the UI. But, it gets the job done :wink:

Continue ReadingImplementing revisioning feature in Open Event