Adding Port Specification for Static File URLs in Open Event Server

Until now, static files stored locally on Open Event server did not have port specification in their URLs. This opened the door for problems while consuming local APIs. This would have created inconsistencies, if two server processes were being served on the same machine but at different ports. In this blog post, I will explain my approach towards solving this problem, and describe code snippets to demonstrate the changes I made in the Open Event Server codebase.

The first part in this process involved finding the source of the bug. For this, my open-source integrated development environment, Microsoft Visual Studio Code turned out to be especially useful. It allowed me to jump from function calls to function definitions quickly:

Screen Shot 2018-08-10 at 12.29.01 PM

I started at events.py and jumped all the way to storage.py, where I finally found out the source of this bug, in upload_local() function:

def upload_local(uploaded_file, key, **kwargs):
    """
    Uploads file locally. Base dir - static/media/
    """
    filename = secure_filename(uploaded_file.filename)
    file_relative_path = 'static/media/' + key + '/' + generate_hash(key) + '/' + filename
    file_path = app.config['BASE_DIR'] + '/' + file_relative_path
    dir_path = file_path.rsplit('/', 1)[0]
    # delete current
    try:
        rmtree(dir_path)
    except OSError:
        pass
    # create dirs
    if not os.path.isdir(dir_path):
        os.makedirs(dir_path)
        uploaded_file.save(file_path)
        file_relative_path = '/' + file_relative_path
    if get_settings()['static_domain']:
        return get_settings()['static_domain'] + \
    file_relative_path.replace('/static', '')
    url = urlparse(request.url)
    return url.scheme + '://' + url.hostname + file_relative_path

Look closely at the return statement:

return url.scheme + '://' + url.hostname + file_relative_path

Bingo! This is the source of our bug. A straightforward solution is to simply concatenate the port number in between, but that will make this one-liner look clumsy – unreadable and un-pythonic. We therefore use Python string formatting:

return '{scheme}://{hostname}:{port}{file_relative_path}'.format(
scheme=url.scheme, hostname=url.hostname, port=url.port,
file_relative_path=file_relative_path)

But this statement isn’t perfect. There’s an edge case that might give unexpected URL. If the port isn’t originally specified, Python’s string formatting heuristic will substitute url.port with None. This will result in a URL like http://localhost:None/some/file_path.jpg, which is obviously something we don’t desire. We therefore append a call to Python’s string replace() method: replace(‘:None’, ”)

The resulting return statement now looks like the following:

return '{scheme}://{hostname}:{port}{file_relative_path}'.format(
scheme=url.scheme, hostname=url.hostname, port=url.port,
file_relative_path=file_relative_path).replace(':None', '')

This should fix the problem. But that’s not enough. We need to ensure that our project adapts well with the change we made. We check this by running the project tests locally:

$ nosetests tests/unittests

Unfortunately, the tests fail with the following traceback:

======================================================================
ERROR: test_create_save_image_sizes (tests.unittests.api.helpers.test_files.TestFilesHelperValidation)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/open-event-server/tests/unittests/api/helpers/test_files.py", line 138, in test_create_save_image_sizes
resized_width_large, _ = self.getsizes(resized_image_file_large)
File "/open-event-server/tests/unittests/api/helpers/test_files.py", line 22, in getsizes
im = Image.open(file)
File "/usr/local/lib/python3.6/site-packages/PIL/Image.py", line 2312, in open
fp = builtins.open(filename, "rb")
FileNotFoundError: [Errno 2] No such file or directory: '/open-event-server:5000/static/media/events/53b8f572-5408-40bf-af97-6e9b3922631d/large/UFNNeW5FRF/5980ede1-d79b-4907-bbd5-17511eee5903.jpg'

It’s evident from this traceback that the code in our test framework is not converting the image url to file path correctly. The port specification part is working fine, but it should not affect file names, they should be independent of port number. The files saved originally do not have port specified in their name, but the code in test framework is expecting port to be involved, hence the above error.

Using the traceback, I went to the code in the test framework where this problem occurred:

def test_create_save_image_sizes(self):
       with app.test_request_context():
           image_url_test = 'https://cdn.pixabay.com/photo/2014/09/08/17/08/hot-air-balloons-439331_960_720.jpg'

           image_sizes_type = "event"
           width_large = 1300
           width_thumbnail = 500
           width_icon = 75
           image_sizes = create_save_image_sizes(image_url_test, image_sizes_type)

           resized_image_url = image_sizes['original_image_url']
           resized_image_url_large = image_sizes['large_image_url']
           resized_image_url_thumbnail = image_sizes['thumbnail_image_url']
           resized_image_url_icon = image_sizes['icon_image_url']

           resized_image_file = app.config.get('BASE_DIR') + resized_image_url.split('/localhost')[1]
           resized_image_file_large = app.config.get('BASE_DIR') + resized_image_url_large.split('/localhost')[1]
           resized_image_file_thumbnail = app.config.get('BASE_DIR') + resized_image_url_thumbnail.split('/localhost')[1]
           resized_image_file_icon = app.config.get('BASE_DIR') + resized_image_url_icon.split('/localhost')[1]

           resized_width_large, _ = self.getsizes(resized_image_file_large)
           resized_width_thumbnail, _ = self.getsizes(resized_image_file_thumbnail)
           resized_width_icon, _ = self.getsizes(resized_image_file_icon)

           self.assertTrue(os.path.exists(resized_image_file))
           self.assertEqual(resized_width_large, width_large)
           self.assertEqual(resized_width_thumbnail, width_thumbnail)
           self.assertEqual(resized_width_icon, width_icon)

 

Obviously, resized_image_url.split(‘/localhost’)[1] will involve port number. So we have to change this line. But this means we also have to change the subsequent lines involving thumbnail, icon and large images. Instead of stripping the port for each of these, we can simply do this collectively at an earlier stage. So we redefine the image_sizes dictionary after the create_save_image_sizes() function call:

image_sizes = {
url_name: urlparse(image_sizes[url_name]).path
for url_name in image_sizes
} # Now file names don't contain port (this gives relative urls).

Now we can simplify the lines each of which earlier required port-stripping code:

resized_image_file = app.config.get('BASE_DIR') + resized_image_url
resized_image_file_large = app.config.get('BASE_DIR') + resized_image_url_large
resized_image_file_thumbnail = app.config.get('BASE_DIR') + resized_image_url_thumbnail
resized_image_file_icon = app.config.get('BASE_DIR') + resized_image_url_icon

We now do a similar modification in test_create_save_resized_image() test method as it also involves URL to file path conversion. We break the line

resized_image_file = app.config.get('BASE_DIR') + resized_image_url.split('/localhost')[1]

to 2 lines:

resized_image_path = urlparse(resized_image_url).path
resized_image_file = app.config.get('BASE_DIR') + resized_image_path

Now let’s run the tests (which failed earlier) again:

Screen Shot 2018-08-10 at 12.29.15 PM.pngFinally, the tests pass without errors! Now, we can add some extra convenience functionality: we can also strip the port when it corresponds with the protocol we’re using. For example, if we’re using https protocol, then we need not specify the port if it is 443, as 443 corresponds to that protocol. We can add this functionality by creating a mapping of such correspondence and checking for it before generating the URL. To do this, we now go back to storage.py and add the following:

SCHEMES = {80: 'http', 443: 'https'}

And add

# No need to specify scheme-corresponding port
port = url.port
if port and url.scheme == SCHEMES.get(url.port, None):
    port = None

just before

return '{scheme}://{hostname}:{port}{file_relative_path}'.format(
scheme=url.scheme, hostname=url.hostname, port=port,
file_relative_path=file_relative_path).replace(':None', '')

And that finishes our work! The tests again pass successfully, plus on top of that we have this new functionality of mentioning ports only when they don’t correspond with the URL scheme!


Resources:

Continue ReadingAdding Port Specification for Static File URLs in Open Event Server

Open Event Server – Export Attendees as CSV File

FOSSASIA‘s Open Event Server is the REST API backend for the event management platform, Open Event. Here, the event organizers can create their events, add tickets for it and manage all aspects from the schedule to the speakers. Also, once he/she makes his event public, others can view it and buy tickets if interested.

The organizer can see all the attendees in a very detailed view in the event management dashboard. He can see the statuses of all the attendees. The possible statuses are completed, placed, pending, expired and canceled, checked in and not checked in. He/she can take actions such as checking in the attendee.

If the organizer wants to download the list of all the attendees as a CSV file, he or she can do it very easily by simply clicking on the Export As and then on CSV.

Let us see how this is done on the server.

Server side – generating the Attendees CSV file

Here we will be using the csv package provided by python for writing the csv file.

import csv
  • We define a method export_attendees_csv which takes the attendees to be exported as a CSV file as the argument.
  • Next, we define the headers of the CSV file. It is the first row of the CSV file.
def export_attendees_csv(attendees):
   headers = ['Order#', 'Order Date', 'Status', 'First Name', 'Last Name', 'Email',
              'Country', 'Payment Type', 'Ticket Name', 'Ticket Price', 'Ticket Type']
  • A list is defined called rows. This contains the rows of the CSV file. As mentioned earlier, headers is the first row.
rows = [headers]
  • We iterate over each attendee in attendees and form a row for that attendee by separating the values of each of the columns by a comma. Here, every row is one attendee.
  • The newly formed row is added to the rows list.
for attendee in attendees:
   column = [str(attendee.order.get_invoice_number()) if attendee.order else '-',
             str(attendee.order.created_at) if attendee.order and attendee.order.created_at else '-',
             str(attendee.order.status) if attendee.order and attendee.order.status else '-',
             str(attendee.firstname) if attendee.firstname else '',
             str(attendee.lastname) if attendee.lastname else '',
             str(attendee.email) if attendee.email else '',
             str(attendee.country) if attendee.country else '',
             str(attendee.order.payment_mode) if attendee.order and attendee.order.payment_mode else '',
             str(attendee.ticket.name) if attendee.ticket and attendee.ticket.name else '',
             str(attendee.ticket.price) if attendee.ticket and attendee.ticket.price else '0',
             str(attendee.ticket.type) if attendee.ticket and attendee.ticket.type else '']

   rows.append(column)
  • rows contains the contents of the CSV file and hence it is returned.
return rows
  • We iterate over each item of rows and write it to the CSV file using the methods provided by the csv package.
writer = csv.writer(temp_file)
from app.api.helpers.csv_jobs_util import export_attendees_csv
content = export_attendees_csv(attendees)
for row in content:
   writer.writerow(row)

Obtaining the Attendees CSV file:

Firstly, we have an API endpoint which starts the task on the server.

GET - /v1/events/{event_identifier}/export/attendees/csv

Here, event_identifier is the unique ID of the event. This endpoint starts a celery task on the server to export the attendees of the event as a CSV file. It returns the URL of the task to get the status of the export task. A sample response is as follows:

{
  "task_url": "/v1/tasks/b7ca7088-876e-4c29-a0ee-b8029a64849a"
}

The user can go to the above-returned URL and check the status of his/her Celery task. If the task completed successfully he/she will get the download URL. The endpoint to check the status of the task is:

and the corresponding response from the server –

{
  "result": {
    "download_url": "/v1/events/1/exports/http://localhost/static/media/exports/1/zip/OGpMM0w2RH/event1.zip"
  },
  "state": "SUCCESS"
}

The file can be downloaded from the above-mentioned URL.

References

Continue ReadingOpen Event Server – Export Attendees as CSV File

Open Event Server – Export Orders as CSV File

FOSSASIA‘s Open Event Server is the REST API backend for the event management platform, Open Event. Here, the event organizers can create their events, add tickets for it and manage all aspects from the schedule to the speakers. Also, once he/she makes his event public, others can view it and buy tickets if interested.

The organizer can see all the orders in a very detailed view in the event management dashboard. He can see the statuses of all the orders. The possible statuses are completed, placed, pending, expired and canceled.

If the organizer wants to download the list of all the orders as a CSV file, he or she can do it very easily by simply clicking on the Export As and then on CSV.

Let us see how this is done on the server.

Server side – generating the Orders CSV file

Here we will be using the csv package provided by python for writing the csv file.

import csv
  • We define a method export_orders_csv which takes the orders to be exported as a CSV file as the argument.
  • Next, we define the headers of the CSV file. It is the first row of the CSV file.
def export_orders_csv(orders):
   headers = ['Order#', 'Order Date', 'Status', 'Payment Type', 'Total Amount', 'Quantity',
              'Discount Code', 'First Name', 'Last Name', 'Email']
  • A list is defined called rows. This contains the rows of the CSV file. As mentioned earlier, headers is the first row.
rows = [headers]
  • We iterate over each order in orders and form a row for that order by separating the values of each of the columns by a comma. Here, every row is one order.
  • The newly formed row is added to the rows list.
for order in orders:
   if order.status != "deleted":
       column = [str(order.get_invoice_number()), str(order.created_at) if order.created_at else '',
                 str(order.status) if order.status else '', str(order.paid_via) if order.paid_via else '',
                 str(order.amount) if order.amount else '', str(order.get_tickets_count()),
                 str(order.discount_code.code) if order.discount_code else '',
                 str(order.user.first_name)
                 if order.user and order.user.first_name else '',
                 str(order.user.last_name)
                 if order.user and order.user.last_name else '',
                 str(order.user.email) if order.user and order.user.email else '']
       rows.append(column)
  • rows contains the contents of the CSV file and hence it is returned.
return rows
  • We iterate over each item of rows and write it to the CSV file using the methods provided by the csv package.
writer = csv.writer(temp_file)
from app.api.helpers.csv_jobs_util import export_orders_csv
content = export_orders_csv(orders)
for row in content:
   writer.writerow(row)

Obtaining the Orders CSV file:

Firstly, we have an API endpoint which starts the task on the server.

GET - /v1/events/{event_identifier}/export/orders/csv

Here, event_identifier is the unique ID of the event. This endpoint starts a celery task on the server to export the orders of the event as a CSV file. It returns the URL of the task to get the status of the export task. A sample response is as follows:

{
  "task_url": "/v1/tasks/b7ca7088-876e-4c29-a0ee-b8029a64849a"
}</span

The user can go to the above-returned URL and check the status of his/her Celery task. If the task completed successfully he/she will get the download URL. The endpoint to check the status of the task is:

and the corresponding response from the server –

{
  "result": {
    "download_url": "/v1/events/1/exports/http://localhost/static/media/exports/1/zip/OGpMM0w2RH/event1.zip"
  },
  "state": "SUCCESS"
}

The file can be downloaded from the aabove-mentionedURL.

References

Continue ReadingOpen Event Server – Export Orders as CSV File

Open Event Server – Export Event as a Pentabarf XML File

FOSSASIA‘s Open Event Server is the REST API backend for the event management platform, Open Event. Here, the event organizers can create their events, add tickets for it and manage all aspects from the schedule to the speakers. Also, once he makes his event public, others can view it and buy tickets if interested.

To make event promotion easier, we also provide the event organizer to export his event as a Pentabarf XML file. Pentabarf XML is used to store events/conferences in a format which most of the scheduling applications can read and add that particular event/conference to the user’s schedule.

Server side – generating the Pentabarf XML file

Here we will be using the pentabarf package for Python for parsing and creating the file.

from pentabarf.Conference import Conference
from pentabarf.Day import Day
from pentabarf.Event import Event
from pentabarf.Person import Person
from pentabarf.Room import Room
  • We define a class PentabarfExporter which has a static method export(event_id).
  • Query the event using the event_id passed and start forming the event in the required format:
event = EventModel.query.get(event_id)
diff = (event.ends_at - event.starts_at)

conference = Conference(title=event.name, start=event.starts_at, end=event.ends_at,
                       days=diff.days if diff.days > 0 else 1,
                       day_change="00:00", timeslot_duration="00:15",
                       venue=event.location_name)
dates = (db.session.query(cast(Session.starts_at, DATE))
        .filter_by(event_id=event_id)
        .filter_by(state='accepted')
        .filter(Session.deleted_at.is_(None))
        .order_by(asc(Session.starts_at)).distinct().all())
  • We have queried for the dates of the event and saved it in dates.
  • We will now iterate over each date and query the microlocations who have a session on that particular date.
for date in dates:
   date = date[0]
   day = Day(date=date)
   microlocation_ids = list(db.session.query(Session.microlocation_id)
                            .filter(func.date(Session.starts_at) == date)
                            .filter_by(state='accepted')
                            .filter(Session.deleted_at.is_(None))
                            .order_by(asc(Session.microlocation_id)).distinct())
  • For each microlocation thus obtained, we will query for accepted sessions to be held at those microlocations.
  • We will also initialize a Room for each microlocation.
for microlocation_id in microlocation_ids:
   microlocation_id = microlocation_id[0]
   microlocation = Microlocation.query.get(microlocation_id)
   sessions = Session.query.filter_by(microlocation_id=microlocation_id) \
       .filter(func.date(Session.starts_at) == date) \
       .filter_by(state='accepted') \
       .filter(Session.deleted_at.is_(None)) \
       .order_by(asc(Session.starts_at)).all()

   room = Room(name=microlocation.name)
  • We will now iterate over the aabove-obtained sessions and instantiate an Event for each session.
  • Then we will iterate over all the speakers of that session and instantiate a Person for each speaker.
  • Finally, we will add that Event to the Room we created earlier.
for session in sessions:

   session_event = Event(id=session.id,
                         date=session.starts_at,
                         start=session.starts_at,
                         duration=str(session.ends_at - session.starts_at) + "00:00",
                         track=session.track.name,
                         abstract=session.short_abstract,
                         title=session.title,
                         type='Talk',
                         description=session.long_abstract,
                         conf_url=url_for('event_detail.display_event_detail_home',
                                          identifier=event.identifier),
                         full_conf_url=url_for('event_detail.display_event_detail_home',
                                               identifier=event.identifier, _external=True),
                         released="True" if event.schedule_published_on else "False")

   for speaker in session.speakers:
       person = Person(id=speaker.id, name=speaker.name)
       session_event.add_person(person)

   room.add_event(session_event)
  • Then we will add the room to the day and then add each day to the conference.
day.add_room(room)
conference.add_day(day)
  • Finally, we will call the generate method of the conference to generate the XML file. This can be directly written to the file.
return conference.generate("Generated by " + get_settings()['app_name'])

Obtaining the Pentabarf XML file:

Firstly, we have an API endpoint which starts the task on the server.

GET - /v1/events/{event_identifier}/export/pentabarf

Here, event_identifier is the unique ID of the event. This endpoint starts a celery task on the server to export the event as a Pentabarf XML file. It returns the task of the URL to get the status of the export task. A sample response is as follows:

{
  "task_url": "/v1/tasks/b7ca7088-876e-4c29-a0ee-b8029a64849a"
}

The user can go to the above-returned URL and check the status of his Celery task. If the task completed successfully he will get the download URL. The endpoint to check the status of the task is:

and the corresponding response from the server –

{
  "result": {
    "download_url": "/v1/events/1/exports/http://localhost/static/media/exports/1/zip/OGpMM0w2RH/event1.zip"
  },
  "state": "SUCCESS"
}

The file can be downloaded from the above-mentioned URL.

Hence, now the event can be added to any scheduling app which recognizes the Pentabarf XML format.

References

Continue ReadingOpen Event Server – Export Event as a Pentabarf XML File

Stripe Authorization in Open Event Server

Stripe is a popular software platform for online payments. Since Open Event  allows the event organizers to sell tickets, an option to accept payments through Stripe is extremely beneficial to the organizer. Stripe allows accepting payments on other’s behalf using Connect. Connect is the Stripe’s full stack solution for platforms that need to process payments and process to multiple parties. This blog post goes over how Event organizers are able to link their Stripe accounts in order to accept payments later.

Registering the platform

The Admin of the Open Event Server will create an account on Stripe and register the platform. Upon creating the  account he/she will get a secret_key and publishable_key.  Moreover on registering the platform a client_id will be provided. These keys are saved in the application settings and only the Admin is authorized to view or change them.

Connecting the Organiser’s account

The Open Event Frontend has a wizard for creating an Event. It provides the organiser an option to connect his/her Stripe account in order to accept payments.

Upon clicking the following button, the organiser is directed to Stripe where he/she can fill the required details.  

The button directs the organizer to the following URL:

https://connect.stripe.com/oauth/authorize?response_type=code&client_id=client_id&scope=read_write&redirect_uri=redirect_uri 

The above URL has the following parameters:

  • client_id – The client ID acquired when registering your platform.required.
  • response_type – Response type. The value is always code. required.
  • redirect_uri – The URL to redirect the customer to after authorization.
  • scope – We need it to be read_write in order to be able to charge on behalf of the customer later.

After successfully entering the required details, the organizer is redirected to the redirect_url as specified in the above URL with a query parameter named as authorization_code. The Frontend sends this code to the Server using the Stripe Authorization endpoint which will be discussed in detail below.

Fetching Tokens from Stripe

The Server accepts the authorization_code by exposing the Stripe Authorization endpoint. It then uses it to fetch organizer’s details and token from Stripe and stores it for future use.

The schema for Stripe Authorization is extremely simple. We require the client to send an authorization_code which will be used to fetch the details. Stripe_publishable_key of the event organizer is exposed via the endpoint and will be used by the Frontend later.

class StripeAuthorizationSchema(Schema):
    """
        Stripe Authorization Schema
    """

    class Meta:
        """
        Meta class for StripeAuthorization Api Schema
        """
        type_ = 'stripe-authorization'
        self_view = 'v1.stripe_authorization_detail'
        self_view_kwargs = {'id': '<id>'}
        inflect = dasherize

    id = fields.Str(dump_only=True)
    stripe_publishable_key = fields.Str(dump_only=True)
    stripe_auth_code = fields.Str(load_only=True, required=True)

    event = Relationship(attribute='event',
                self_view='v1.stripe_authorization_event',
                self_view_kwargs={'id': '<id>'},
                related_view='v1.event_detail',
                related_view_kwargs={'stripe_authorization_id':                      '<id>'},
                schema="EventSchema",
                type_='event')

We use the Requests library in order to fetch the results. First we fetch the client_id that we had stored in the application settings using a helper method called get_credentials. We then use it along with the authorization_code in order to make a POST request to Stripe Connect API. The full method is given below for reference.

@staticmethod
def get_event_organizer_credentials_from_stripe(stripe_auth_code):
        """
        Uses the stripe_auth_code to get the other credentials for the event organizer's stripe account
        :param stripe_auth_code: stripe authorization code
        :return: response from stripe
        """
        credentials = StripePaymentsManager.get_credentials()

        if not credentials:
            raise Exception('Stripe is incorrectly configured')

        data = {
            'client_secret': credentials['SECRET_KEY'],
            'code': stripe_auth_code,
            'grant_type': 'authorization_code'
        }

        response = requests.post('https://connect.stripe.com/oauth/token', data=data)
        return json.loads(response.text)

We call the above method before creating the object using the before_create_object method of Marshmallow which allows us to do data preprocessing and validations.

If the request was a success, the response from Stripe connect API includes all the details necessary to accept payments on their behalf. We add these fields to the data and save it in the database.

{
  "token_type": "bearer",
  "stripe_publishable_key": PUBLISHABLE_KEY,
  "scope": "read_write",
  "livemode": false,
  "stripe_user_id": USER_ID,
  "refresh_token": REFRESH_TOKEN,
  "access_token": ACCESS_TOKEN
}

In case there was an error, an error_description would be returned. This error_description is sent back to the frontend and shown to the event organizer.

{
  "error": "invalid_grant",
  "error_description": "Authorization code already used:                                               
                        AUTHORIZATION_CODE"
}

After successfully fetching the results, we save it inside the database and return the stripe_publishable_key which will be used by the Frontend when charging the ticket buyers later.

Lastly we can go over the Stripe Authorization model as well. The stripe_secret_key will be used to charge the customers later.

id = db.Column(db.Integer, primary_key=True)
stripe_secret_key = db.Column(db.String)
stripe_refresh_token = db.Column(db.String)
stripe_publishable_key = db.Column(db.String)
stripe_user_id = db.Column(db.String)
stripe_auth_code = db.Column(db.String)

References

 

Continue ReadingStripe Authorization in Open Event Server

Open Event Server – Export Event as xCalendar File

FOSSASIA‘s Open Event Server is the REST API backend for the event management platform, Open Event. Here, the event organizers can create their events, add tickets for it and manage all aspects from the schedule to the speakers. Also, once he makes his event public, others can view it and buy tickets if interested.

To make event promotion easier, we also provide the event organizer to export his event as an xCalendar file. xCal is an XML representation of the iCalendar standard. xCal is not an alternative nor next generation of iCalendar. xCal represents iCalendar components, properties, and parameters as defined in iCalendar. This format was selected to ease its translation back to the iCalendar format using an XSLT transform.

Server side – generating the xCal file

Here we will be using the xml.etree.ElementTree package for Python for parsing and creating XML data.

from xml.etree.ElementTree import Element, SubElement, tostring
  • We define a class XCalExporter which has a static method export(event_id).
  • Query the event using the event_id passed and start forming the calendar:
event = Event.query.get(event_id)

tz = event.timezone or 'UTC'
tz = pytz.timezone(tz)

i_calendar_node = Element('iCalendar')
i_calendar_node.set('xmlns:xCal', 'urn:ietf:params:xml:ns:xcal')
v_calendar_node = SubElement(i_calendar_node, 'vcalendar')
version_node = SubElement(v_calendar_node, 'version')
version_node.text = '2.0'
prod_id_node = SubElement(v_calendar_node, 'prodid')
prod_id_node.text = '-//fossasia//open-event//EN'
cal_desc_node = SubElement(v_calendar_node, 'x-wr-caldesc')
cal_desc_node.text = "Schedule for sessions at " + event.name
cal_name_node = SubElement(v_calendar_node, 'x-wr-calname')
cal_name_node.text = event.name
  • We query for the accepted sessions of the event and store it in sessions
sessions = Session.query \
   .filter_by(event_id=event_id) \
   .filter_by(state='accepted') \
   .filter(Session.deleted_at.is_(None)) \
   .order_by(asc(Session.starts_at)).all()
  • We then iterate through all the sessions in sessions.
  • If it is a valid session, we instantiate a SubElement and store required details
v_event_node = SubElement(v_calendar_node, 'vevent')

method_node = SubElement(v_event_node, 'method')
method_node.text = 'PUBLISH'

uid_node = SubElement(v_event_node, 'uid')
uid_node.text = str(session.id) + "-" + event.identifier

dtstart_node = SubElement(v_event_node, 'dtstart')
dtstart_node.text = tz.localize(session.starts_at).isoformat()

…. So on
  • We then loop through all the speakers in that particular session and add it to the xCal calendar node object as well.
for speaker in session.speakers:
   attendee_node = SubElement(v_event_node, 'attendee')
   attendee_node.text = speaker.name
  • And finally, the string of the calendar node is returned. This is the xCalendar file contents. This can be directly written to a file.
return tostring(i_calendar_node)

Obtaining the xCal file:

Firstly, we have an API endpoint which starts the task on the server.

GET - /v1/events/{event_identifier}/export/xcal

Here, event_identifier is the unique ID of the event. This endpoint starts a celery task on the server to export the event as an xCal file. It returns the URL of the task to get the status of the export task. A sample response is as follows:

{
  "task_url": "/v1/tasks/b7ca7088-876e-4c29-a0ee-b8029a64849a"
}

The user can go to the above-returned URL and check the status of his Celery task. If the task completed successfully he will get the download URL. The endpoint to check the status of the task is:

and the corresponding response from the server –

{
  "result": {
    "download_url": "/v1/events/1/exports/http://localhost/static/media/exports/1/zip/OGpMM0w2RH/event1.zip"
  },
  "state": "SUCCESS"
}

The file can be downloaded from the above mentioned URL.

Hence, now the event can be added to any scheduling app which recognizes the xcs format.

References

Continue ReadingOpen Event Server – Export Event as xCalendar File

Open Event Server – Export Event as an iCalendar File

FOSSASIA‘s Open Event Server is the REST API backend for the event management platform, Open Event. Here, the event organizers can create their events, add tickets for it and manage all aspects from the schedule to the speakers. Also, once he makes his event public, others can view it and buy tickets if interested.

To make event promotion easier, we also provide the event organizer to export his event as an iCalendar file. Going by the Wikipedia definition, iCalendar is a computer file format which allows Internet users to send meeting requests and tasks to other Internet users by sharing or sending files in this format through various methods. The files usually have an extension of .ics. With supporting software, such as an email reader or calendar application, recipients of an iCalendar data file can respond to the sender easily or counter propose another meeting date/time. The file format is specified in a proposed internet standard (RFC 5545) for calendar data exchange.

Server side – generating the iCal file

Here we will be using the icalendar package for Python as the file writer.

from icalendar import Calendar, vCalAddress, vText
  • We define a class ICalExporter which has a static method export(event_id).
  • Query the event using the event_id passed and start forming the calendar:
event = EventModel.query.get(event_id)

cal = Calendar()
cal.add('prodid', '-//fossasia//open-event//EN')
cal.add('version', '2.0')
cal.add('x-wr-calname', event.name)
cal.add('x-wr-caldesc', "Schedule for sessions at " + event.name)
  • We query for the accepted sessions of the event and store it in sessions.
sessions = Session.query \
   .filter_by(event_id=event_id) \
   .filter_by(state='accepted') \
   .filter(Session.deleted_at.is_(None)) \
   .order_by(asc(Session.starts_at)).all()
  • We then iterate through all the sessions in sessions.
  • If it is a valid session, we instantiate an icalendar event and store required details.
event_component = icalendar.Event()
event_component.add('summary', session.title)
event_component.add('uid', str(session.id) + "-" + event.identifier)
event_component.add('geo', (event.latitude, event.longitude))
event_component.add('location', session.microlocation.name or '' + " " + event.location_name)
event_component.add('dtstart', tz.localize(session.starts_at))
event_component.add('dtend', tz.localize(session.ends_at))
event_component.add('email', event.email)
event_component.add('description', session.short_abstract)
event_component.add('url', url_for('event_detail.display_event_detail_home',
                                  identifier=event.identifier, _external=True))
  • We then loop through all the speakers in that particular session and add it to the iCal Event object as well.
for speaker in session.speakers:
   # Ref: http://icalendar.readthedocs.io/en/latest/usage.html#file-structure
   # can use speaker.email below but privacy reasons
   attendee = vCalAddress('MAILTO:' + event.email if event.email else 'undefined@email.com')
   attendee.params['cn'] = vText(speaker.name)
   event_component.add('attendee', attendee)
  • This event_component is then added to the cal object that we created in the beginning.
cal.add_component(event_component)
  • And finally, the cal.to_ical() is returned. This is the iCalendar file contents. This can be directly written to a file.
return cal.to_ical()

Obtaining the iCal file:

Firstly, we have an API endpoint which starts the task on the server.

GET - /v1/events/{event_identifier}/export/ical

Here, event_identifier is the unique ID of the event. This endpoint starts a celery task on the server to export the event as an iCal file. It returns the task of the URL to get the status of the export task. A sample response is as follows:

{
  "task_url": "/v1/tasks/b7ca7088-876e-4c29-a0ee-b8029a64849a"
}

The user can go to the above returned URL and check the status of his Celery task. If the task completed successfully he will get the download URL. The endpoint to check the status of the task is:

and the corresponding response from the server –

{
  "result": {
    "download_url": "/v1/events/1/exports/http://localhost/static/media/exports/1/zip/OGpMM0w2RH/event1.zip"
  },
  "state": "SUCCESS"
}

The file can be downloaded from the above mentioned URL.

Hence, now the event can be added to any scheduling app which recognizes the ics format.

References

Continue ReadingOpen Event Server – Export Event as an iCalendar File

Update of Python Runtime in Meilix

Meilix Generator is a webapp uses flask with Python, Werkzeug, and Jinja 2. It triggers Travis to release in an ISO. It is deployed on Heroku. An older python version caused the webapp to load very slowly and it also become unsupported so there was a need to update the python version. In this blog post we walk through the update of Python in the project.

We can specify an explicit version of Python to be used to run your application. For example, if you require Python 2, add the following to your Pipfile:

[requires]
Python_version = "2.7"

 

Then run $ pipnv lock to generate Pipfile.lock and push to Heroku.

Another way:

If we are using pip, we can supply a runtime.txt file.

$ cat runtime.txt
python-3.6.1

 

Building of the webapp (Example)

The webapp build in Heroku and provide us a log. The log presents the packages installed and its version. Log also shows if any newer version is present for the package.

While building webapp, we get this as a log:

-----> Python app detected
! The latest version of Python 3 is python-3.6.5 (you are using python-3.6.1, which is unsupported).
! We recommend upgrading by specifying the latest version (python-3.6.5).

 

This confirms that we need to update python version and so thus we edited the runtime.txt

Now building the same webapp, we get:

Python app detected
-----> Found python-3.6.1, removing
-----> Installing python-3.6.5
-----> Installing pip

 

It already using older python, so it need first to remove the older version and then it install the latest one.

The same implementation can be seen in the history.

Reference:

Flask – The Microframework

Heroku Python Runtime

Continue ReadingUpdate of Python Runtime in Meilix

Adding Features into Meilix Generator Webapp

Meilix Generator is a webapp generated in FOSSASIA which takes input from user and send it to Meilix to trigger a build. Then a release is made whose link is emailed to the user. The webapp contains a form where there are fields for installing a particular packages, etc. In the following we will discuss about the ways to achieve the configurations in Meilix ISO without even touching the Meilix repo.

For adding an option in the webapp:

Editing the frontend

We need to edit this line in the index.html with this line:

<input name = "GENERATOR_package_vlc" type = "checkbox" value = "vlc" id = "vlc">

 

Making the script

Then we have to add a script in the script folder. This script is called by Meilix build script. This script contains the variable “GENERATOR_package_vlc”.

We name this file vlc-package.sh

Content:

#!/bin/bash
if echo "$GENERATOR_package_vlc" | grep -q vlc; then 
sudo apt-get install -q -y vlc; fi

 

Line 2 checks that the vlc is checked in the checkbox or not, if it is checked then the other next line gets executed otherwise not.

Calling the script from Meilix (needs to be done only once)

We will add a line in the Meililx build script to call those script present in the Meilix Generator repo.

SCRIPT_URL=https://www.github.com/fossasia/meilix-generator/archive/master.zip
wget -O $scripts.zip $SCRIPT_URL
unzip scripts.zip
SCRIPTS_FOLDER_IN_ZIP="meilix-generator-master/scripts"
ls $SCRIPTS_FOLDER_IN_ZIP; do
$SCRIPTS_FOLDER_IN_ZIP/script; done			#execute all scripts

 

Setting the URL via travis build config post to get all the values starting with GENERATOR_

GENERATOR_ = request.form['GENERATOR_']

 

So overall the abstract of the idea is:

  1. Getting the variables from html to travis as environment variable
  2. Cloning the meilix repo
  3. Executing all the scripts present.

References:

Request HTTP Python

Online Installation media

 

Continue ReadingAdding Features into Meilix Generator Webapp

Open Event Server – Pages API

This article illustrates how the Pages API has been designed and implemented on the server side, i.e., FOSSASIA‘s Open Event Server. Pages endpoint is used to create static pages such as “About Page” or any other page that doesn’t need to be updated frequently and only a specific content is to be shown.

Parameters

  1. name – This stores the name of the page.
      1. Type – String
      2. Required – Yes
  2. title – This stores the title of the page.
      1. Type – String
      2. Required – No
  3. url – This stores the url of the page.
      1. Type – String
      2. Required – Yes
  4. description – This stores the description of the page.
      1. Type – String
      2. Required – Yes
  5. language – This stores the language of the page.
      1. Type – String
      2. Required – No
  6. index – This stores the position of the page.
      1. Type – Integer
      2. Required – No
      3. Default – 0
  7. place – Location where the page will be placed.
      1. Type – String
      2. Required – No
      3. Accepted Values – ‘footer’ and ‘event’

These are the allowed parameters for the endpoint.

Model

Lets see how we model this API. The ORM looks like this :

__tablename__ = 'pages'
id = db.Column(db.Integer, primary_key=True)
name = db.Column(db.String, nullable=False)
title = db.Column(db.String)
url = db.Column(db.String, nullable=False)
description = db.Column(db.String)
place = db.Column(db.String)
language = db.Column(db.String)
index = db.Column(db.Integer, default=0)

As you can see, we created a table called “pages”. This table has 8 columns, 7 of which are the parameters that I have mentioned above. The column “id” is an Integer column and is the primary key column. This will help to differentiate between the various entries in the table.

The visualisation for this table looks as follows :

API

We support the following operations:

  1. GET all the pages in the database
  2. POST create a new page
  3. GET details of a single page as per id
  4. PATCH a single page by id
  5. DELETE a single page by id

To implement this we first add the routes in our python file as follows :

api.route(PageList, 'page_list', '/pages')
api.route(PageDetail, 'page_detail', '/pages/<int:id>')

Then we define these classes to handle the requests. The first route looks as follows:

class PageList(ResourceList):
   """
   List and create page
   """
   decorators = (api.has_permission('is_admin', methods="POST"),)
   schema = PageSchema
   data_layer = {'session': db.session,
                 'model': Page}

As can be seen above, this request requires the user to be an admin. It uses the Page model described above and handles a POST request.

The second route is:

class PageDetail(ResourceDetail):
   """
   Page detail by id
   """
   schema = PageSchema
   decorators = (api.has_permission('is_admin', methods="PATCH,DELETE"),)
   data_layer = {'session': db.session,
                 'model': Page}

This route also requires the user to be an admin. It uses the Page model and handles PATCH, DELETE requests.

To summarise our APIs are:

GET

/v1/pages{?sort,filter}

POST

/v1/pages{?sort,filter}

GET

/v1/pages/{page_id}

PATCH

/v1/pages/{page_id}

DELETE

/v1/pages/{page_id}

References

Continue ReadingOpen Event Server – Pages API