Implementing Scheduler Actions on Open Event Frontend

After the functionality to display scheduled sessions was added to Open Event Frontend, the read-only implementation of the scheduler had been completed. What was remaining now in the scheduler were the write actions, i.e., the sessions’ scheduling which event organizers do by deciding its timings, duration and venue.

First of all, these actions required the editable flag to be true for the fullcalendar plugin. This allowed the sessions displayed to be dragged and dropped. Once this was enabled, the next task was to embed data in each of the unscheduled sessions so that when they get dropped on the fullcalendar space, they get recognized by the calendar, which can place it at the appropriate location. For this functionality, they had to be jQuery UI draggables and contain an “event” data within them. This was accomplished by the following code:

this.$().draggable({
  zIndex         : 999,
  revert         : true,      // will cause the event to go back to its
  revertDuration : 0  //  original position after the drag
});

this.$().data('event', {
  title    : this.$().text().replace(/\s\s+/g, ' '), // use the element's text as the event title
  id       : this.$().attr('id'),
  serverId : this.get('session.id'),
  stick    : true, // maintain when user navigates (see docs on the renderEvent method)
  color    : this.get('session.track.color')
});

Here, “this” refers to each unscheduled session. Note that the session color is fetched via the corresponding session track. Once the unscheduled sessions contain enough relevant data and are of the right type (i.e, jQuery UI draggable type), they’re ready to be dropped on the fullcalendar space.

Now, when an unscheduled session is dropped on the fullcalendar space, fullcalendar’s eventReceive callback is triggered after its drop callback. In this callback, the code removes the session data from the unscheduled sessions’ list, so it disappears from there and gets stuck to the fullcalendar space. Then the code in the drop callback makes a PATCH request to Open Event Server with the relevant data, i.e, start and end times as well as microlocation. This updates the corresponding session on the server.

Similarly, another callback is generated when an event is resized, which means when its duration is changed. This again sends a corresponding session PATCH request to the server. Furthermore, the functionality to pop a scheduled event out of the calendar and add it back to the unscheduled sessions’ list is also implemented, just like in Eventyay version 1. For this, a cross button is implemented, which is embedded in each scheduled session. Clicking this pops the session out of the calendar and adds it back to the unscheduled sessions list. Again, a corresponding PATCH request is sent to the server.

After getting the response of such requests, a notification is displayed on the screen, which informs the users whether the action was successful or not. The main PATCH functionality is in a separate function which is called by different callbacks accordingly, so code reusability is increased:

updateSession(start, end, microlocationId, sessionId) {
    let payload = {
      data: {
        attributes: {
          'starts-at' : start ? start.toISOString() : null,
          'ends-at'   : end ? end.toISOString() : null
        },
        relationships: {
          microlocation: {
            data: {
              type : 'microlocation',
              id   : microlocationId
            }
          }
        },
        type : 'session',
        id   : sessionId
      }
    };

    let config = {
      skipDataTransform: true
    };
    return this.get('loader')
      .patch(`sessions/${sessionId}`, JSON.stringify(payload), config)
      .then(() => {
        this.get('notify').success('Changes have been made successfully');
      })
      .catch(reason => {
        this.set('error', reason);
        this.get('notify').error(`Error: ${reason}`);
      });
  },

This completes the scheduler implementation on Open Event Frontend. Here is how it looks in action:

scheduler actions.gif

Resources

Continue Reading

Open Event Server – Export Speakers as PDF File

FOSSASIA‘s Open Event Server is the REST API backend for the event management platform, Open Event. Here, the event organizers can create their events, add tickets for it and manage all aspects from the schedule to the speakers. Also, once he/she makes his event public, others can view it and buy tickets if interested.

The organizer can see all the speakers in a very detailed view in the event management dashboard. He can see the statuses of all the speakers. The possible statuses are pending, accepted, and rejected. He/she can take actions such as editing the speakers.

If the organizer wants to download the list of all the speakers as a PDF file, he or she can do it very easily by simply clicking on the Export As PDF button in the top right-hand corner.

Let us see how this is done on the server.

Server side – generating the Speakers PDF file

Here we will be using the pisa package which is used to convert from HTML to PDF. It is a html2pdf converter which uses ReportLab Toolkit, the HTML5lib and pyPdf. It supports HTML5 and CSS 2.1 (and some of CSS 3). It is completely written in pure Python so it is platform independent.

from xhtml2pdf import pisa<

We have a utility method create_save_pdf which creates and saves PDFs from HTML. It takes the following arguments:

  • pdf_data – This contains the HTML template which has to be converted to PDF.
  • key – This contains the file name
  • dir_path – This contains the directory

It returns the newly formed PDF file. The code is as follows:

def create_save_pdf(pdf_data, key, dir_path='/static/uploads/pdf/temp/'):
   filedir = current_app.config.get('BASE_DIR') + dir_path

   if not os.path.isdir(filedir):
       os.makedirs(filedir)

   filename = get_file_name() + '.pdf'
   dest = filedir + filename

   file = open(dest, "wb")
   pisa.CreatePDF(io.BytesIO(pdf_data.encode('utf-8')), file)
   file.close()

   uploaded_file = UploadedFile(dest, filename)
   upload_path = key.format(identifier=get_file_name())
   new_file = upload(uploaded_file, upload_path)
   # Removing old file created
   os.remove(dest)

   return new_file

The HTML file is formed using the render_template method of flask. This method takes the HTML template and its required variables as the arguments. In our case, we pass in ‘pdf/speakers_pdf.html’(template) and speakers. Here, speakers is the list of speakers to be included in the PDF file. In the template, we loop through each item of speakers. We print his name, email, list of its sessions, mobile, a short biography, organization, and position. All these fields form a row in the table. Hence, each speaker is a row in our PDF file.

The various columns are as follows:

<thead>
<tr>
   <th>
       {{ ("Name") }}
   </th>
   <th>
       {{ ("Email") }}
   </th>
   <th>
       {{ ("Sessions") }}
   </th>
   <th>
       {{ ("Mobile") }}
   </th>
   <th>
       {{ ("Short Biography") }}
   </th>
   <th>
       {{ ("Organisation") }}
   </th>
   <th>
       {{ ("Position") }}
   </th>
</tr>
</thead>

A snippet of the code which handles iterating over the speakers’ list and forming a row is as follows:

{% for speaker in speakers %}
   <tr class="padded" style="text-align:center; margin-top: 5px">
       <td>
           {% if speaker.name %}
               {{ speaker.name }}
           {% else %}
               {{ "-" }}
           {% endif %}
       </td>
       <td>
           {% if speaker.email %}
               {{ speaker.email }}
           {% else %}
               {{ "-" }}
           {% endif %}
       </td>
       <td>
           {% if speaker.sessions %}
               {% for session in speaker.sessions %}
                   {{ session.name }}<br>
               {% endfor %}
           {% else %}
               {{ "-" }}
           {% endif %}
       </td>
      …. So on
   </tr>
{% endfor %}

The full template can be found here.

Obtaining the Speakers PDF file:

Firstly, we have an API endpoint which starts the task on the server.

GET - /v1/events/{event_identifier}/export/speakers/pdf

Here, event_identifier is the unique ID of the event. This endpoint starts a celery task on the server to export the speakers of the event as a PDF file. It returns the URL of the task to get the status of the export task. A sample response is as follows:

{
  "task_url": "/v1/tasks/b7ca7088-876e-4c29-a0ee-b8029a64849a"
}

The user can go to the above-returned URL and check the status of his/her Celery task. If the task completed successfully he/she will get the download URL. The endpoint to check the status of the task is:

and the corresponding response from the server –

{
  "result": {
    "download_url": "/v1/events/1/exports/http://localhost/static/media/exports/1/zip/OGpMM0w2RH/event1.zip"
  },
  "state": "SUCCESS"
}

The file can be downloaded from the above-mentioned URL.

Resources

Continue Reading

Implementing Unscheduled Sessions List for Event Scheduler

Until recently, Open Event Server didn’t allow the storage of unscheduled sessions. However, having the provision of unscheduled sessions was necessary so that event organizers can easily schedule the unscheduled sessions and keep track of them. Also, it allows them to remove scheduled sessions from the scheduler and place them in the unscheduled sessions list, so that they can be scheduled later. Also, since the unscheduled sessions list was also present in Eventyay version 1, it was decided to have the same in version 2.

The first step was to enable the storage of unscheduled sessions on the server. For this, the starts-at and ends-at fields of the session model were modified to be non-required (earlier they were mandatory). Once this change was done, the next step was to fetch the list of unscheduled sessions on the frontend, from the server. Unscheduled sessions were the ones which had the starts-at and ends-at fields as null. Also, the speakers’ details needed to be fetched so that their names can be mentioned along with sessions’ titles, in accordance with Eventyay version 1. Thus, the following were the filter options for the unscheduled sessions’ fetching:

let unscheduledFilterOptions = [
      {
        and: [
          {
            name : 'starts-at',
            op   : 'eq',
            val  : null
          },
          {
            name : 'ends-at',
            op   : 'eq',
            val  : null
          }
        ]
      }
];
 
let unscheduledSessions = await eventDetails.query('sessions', {
      include : 'speakers,track',
      filter  : unscheduledFilterOptions
    });

 

This gave us the list of unscheduled sessions on the frontend appropriately. After this, the next step was to display this list to the event organizer. For this, the scheduler’s Handlebars template file was modified appropriately. The colors and sizes were chosen so that the list looks similar to the one in Eventyay version 1. Also, the Ember add-on named ember-drag-drop was used to make these unscheduled session components draggable, so that they can be ultimately scheduled on the scheduler. After installing this package and making the necessary changes to the project’s package.json file, the component file for unscheduled sessions was modified accordingly to adapt for the draggable components’ UI. This was the final step and completed the implementation of listing unscheduled sessions.

unscheduled_sessions.gif

Resources

Continue Reading

Open Event Frontend – Events Explore Page

This blog illustrates how the events explore page is implemented in the Open Event Frontend web app. The user can land on the events explore page by clicking on Browse Events button in the top panel on the home page, shown by the mouse tip in the following picture.

Here, the user can use the various filter options provided to search for the events as per his requirements, He/she can filter according to categories, sub-categories for each category, event type, and date range. A unique feature here is that the user can pick from the start date range options such as today, tomorrow, this week, this weekend, next week and many more. If neither of these fits his needs he can use custom dates as well. The user can also filter events using event location which is autocompleted using Google Maps API. Thus, searching for events is fast, easy and fun.

Let us see how this has been implemented.

Implementation

The explore routes has a method _loadEvents(params). Here, params is the various query parameters for filtering the events. This method forms the query, sends it to the server and returns the list of events returned by the server. The server uses Flask-REST-JSONAPI. It has a very flexible filtering system. It is completely related to the data layer used by the ResourceList manager. More information about this can be found here.

So, the filters are formed using syntax specified in the link mentioned above. We form an array filterOptions which stores the various filters. The default filter is that the event should be published:

let filterOptions = [
 {
   name : 'state',
   op  : 'eq',
   val  : 'published'
 }
];

Then we check for each filter option and check if it is present or not. If yes then we add it to filterOptions. An example as follows:

if (params.category) {
 filterOptions.push({
   name : 'event-topic',
   op  : 'has',
   val  : {
     name : 'name',
     op : 'eq',
     val : params.category
   }
 });
}

This is repeated for sub_category, event_type, location and start_date and end_date. An event is considered to fulfill the date filter if it satisfies any one of the given conditions:

  • If both start_date and end_date are mentioned:
    • Event start_date is after filter start date and before filter end date.
    • Or, event end date if after filter start date and before filter end date.
    • Or, event start date is before filter start date and event end date date is after filter end date.
  • If only start_date is mentioned, then if the event start date is after filter start date or event end date is after filter start date.

The code to this can be found here. For the date ranges mentioned above(today, tomorrow etc) the start dates and end dates are calculated using the moment.js library and then passed on as params.

The filteredEvents are passed in the route model.

async model(params) {
 return {
   eventTypes     : await this.store.findAll('event-type'),
   eventTopics    : await this.store.findAll('event-topic', { include: 'event-sub-topics' }),
   filteredEvents : await this._loadEvents(params)
 };
}

The variable is set in the controller and any change to the query params is observed for. The method _loadEvents is called whenever the query params change.

setupController(controller, model) {
 this._super(...arguments);
 controller.set('filteredEvents', model.filteredEvents);
 this.set('controller', controller);
},

actions: {
 async queryParamsDidChange(change, params) {
   if (this.get('controller')) {
     this.get('controller').set('filteredEvents', await this._loadEvents(params));
   }
 }
}

The template iterates over the filteredEvents and displays each one in a card.

Resources

Continue Reading

Open Event Server – Export Sessions as PDF File

FOSSASIA‘s Open Event Server is the REST API backend for the event management platform, Open Event. Here, the event organizers can create their events, add tickets for it and manage all aspects from the schedule to the speakers. Also, once he/she makes his event public, others can view it and buy tickets if interested.

The organizer can see all the sessions in a very detailed view in the event management dashboard. He can see the statuses of all the sessions. The possible statuses are pending, accepted, confirmed and rejected. He/she can take actions such as accepting/rejecting the sessions.

If the organizer wants to download the list of all the sessions as a PDF file, he or she can do it very easily by simply clicking on the Export As PDF button in the top right-hand corner.

Let us see how this is done on the server.

Server side – generating the Sessions PDF file

Here we will be using the pisa package which is used to convert from HTML to PDF. It is a html2pdf converter which uses ReportLab Toolkit, the HTML5lib and pyPdf. It supports HTML5 and CSS 2.1 (and some of CSS 3). It is completely written in pure Python so it is platform independent.

from xhtml2pdf import pisa

We have a utility method create_save_pdf which creates and saves PDFs from HTML. It takes the following arguments:

  • pdf_data – This contains the HTML template which has to be converted to PDF.
  • key – This contains the file name
  • dir_path – This contains the directory

It returns the newly formed PDF file. The code is as follows:

def create_save_pdf(pdf_data, key, dir_path='/static/uploads/pdf/temp/'):
   filedir = current_app.config.get('BASE_DIR') + dir_path

   if not os.path.isdir(filedir):
       os.makedirs(filedir)

   filename = get_file_name() + '.pdf'
   dest = filedir + filename

   file = open(dest, "wb")
   pisa.CreatePDF(io.BytesIO(pdf_data.encode('utf-8')), file)
   file.close()

   uploaded_file = UploadedFile(dest, filename)
   upload_path = key.format(identifier=get_file_name())
   new_file = upload(uploaded_file, upload_path)
   # Removing old file created
   os.remove(dest)

   return new_file

The HTML file is formed using the render_template method of flask. This method takes the HTML template and its required variables as the arguments. In our case, we pass in ‘pdf/sessions_pdf.html’(template) and sessions. Here, sessions is the list of sessions to be included in the PDF file. In the template, we loop through each item of sessions and check if it is deleted or not. If it not deleted then we print its title, state, list of its speakers, track, created at and has an email been sent or not. All these fields form a row in the table. Hence, each session is a row in our PDF file.

The various columns are as follows:

<thead>
<tr>
   <th>
       {{ ("Title") }}
   </th>
   <th>
       {{ ("State") }}
   </th>
   <th>
       {{ ("Speakers") }}
   </th>
   <th>
       {{ ("Track") }}
   </th>
   <th>
       {{ ("Created At") }}
   </th>
   <th>
       {{ ("Email Sent") }}
   </th>
</tr>
</thead>

A snippet of the code which handles iterating over the sessions list and forming a row is as follows:

{% for session in sessions %}
   {% if not session.deleted_at %}
       <tr class="padded" style="text-align:center; margin-top: 5px">
           <td>
               {% if session.title %}
                   {{ session.title }}
               {% else %}
                   {{ "-" }}
               {% endif %}
           </td>
           <td>
               {% if session.state %}
                   {{ session.state }}
               {% else %}
                   {{ "-" }}
               {% endif %}
           </td>
           <td>
               {% if session.speakers %}
                   {% for speaker in session.speakers %}
                       {{ speaker.name }}<br>
                   {% endfor %}
               {% else %}
                   {{ "-" }}
               {% endif %}
           </td>
          ….. And so on
       </tr>
   {% endif %}
{% endfor %}

The full template can be found here.

Obtaining the Sessions PDF file:

Firstly, we have an API endpoint which starts the task on the server.

GET - /v1/events/{event_identifier}/export/sessions/pdf

Here, event_identifier is the unique ID of the event. This endpoint starts a celery task on the server to export the sessions of the event as a PDF file. It returns the URL of the task to get the status of the export task. A sample response is as follows:

{
  "task_url": "/v1/tasks/b7ca7088-876e-4c29-a0ee-b8029a64849a"
}

The user can go to the above-returned URL and check the status of his/her Celery task. If the task completed successfully he/she will get the download URL. The endpoint to check the status of the task is:

and the corresponding response from the server –

{
  "result": {
    "download_url": "/v1/events/1/exports/http://localhost/static/media/exports/1/zip/OGpMM0w2RH/event1.zip"
  },
  "state": "SUCCESS"
}

The file can be downloaded from the above-mentioned URL.

Resources

Continue Reading

Implementing User Email Verification in Open Event Frontend

Open Event Server provides the functionality of user email verification after a user registers, but it was not implemented on Open Event Frontend until recently. For users, this meant they were still not able to verify themselves, even after receiving confirmation links in their inboxes, which were sent by the server. Thus, implementing it on frontend was crucial for a complete user registration workflow.

Since the server had already exposed an endpoint to perform the registration, all that was required on the frontend to be done was to make a call to this endpoint with the necessary data. The entire process can be summarized as follows:

  1. The recently registered user clicks on the verification link she receives on her email
  2. The above step opens the link, which is of the format http://fossasia.github.io/open-event-frontend/verify?token=
  3. As soon as the frontend server receives this request, it extracts the token from the URL query parameter
  4. The token is now sent to the backend server as a patch request
  5. The response of the above request confirms whether the user verification is successful or not, and an appropriate message is displayed

In the frontend code, the above algorithm is spread across 3 files: the router, verify route and verify controller. A new route named /verify was implemented for the user verification, and was registered in the project’s main router.js file. After that, in the verify route, the beforeModel() method is used to trigger the above algorithm before the page is loaded:

// in app/routes/verify.js

beforeModel(transition) {
this.controllerFor('verify').verify(transition.queryParams.token);
}

The main algorithm above is implemented in the verify controller:

// in app/controllers/verify.js
...
queryParams : ['token'],
token       : null,
success     : false,
error       : null,

verify(tokenVal) {
let payload = {
data: {
token: tokenVal
}
};
return this.get('loader')
.post('auth/verify-email', payload)
.then(() => {
this.set('success', true);
})
.catch(reason => {
this.set('error', reason);
this.set('success', false);
});
}
});

 

A template for displaying the success or failure messages to the user was also created. It uses the value of the success boolean set above to decide the message to be displayed to the user. The user registration workflow is now complete and the user sees the following message after clicking on the verification link she receives:

Screen Shot 2018-08-21 at 7.10.43 PM

Resources

Continue Reading

Upgrading Open Event to Use Sendgrid API v3

Sendgrid recently upgraded their web API to send emails, and support for previous versions was deprecated. As a result, Open Event Server’s mail sending tasks were rendered unsuccessful, because the requests they were sending to Sendgrid were not being processed. On top of that, it was also found out later that the existing Sendgrid API key on the development server was expired. This had to be fixed at the earliest because emails are a core part of Open Event functionality.

The existing way for emails to be sent via Sendgrid used to hit the endpoint “https://api.sendgrid.com/api/mail.send.json” to send emails. Also, the payload structure was as follows:

payload = {
    'to': to,
    'from': email_from,
    'subject': subject,
    'html': html
}

Also, a header  “Authorization”: “Bearer ” accompanied the above payload. However, Sendgrid changed the payload structure to be of the following format:

{

“personalizations”: [

{“to”: [

{“email”: “[email protected]“}

]

}

],

“from”: {

“email”: “[email protected]

},

“subject”: “Hello, World!”,

“content”: [

{

“type”: “text/plain”,

“value”: “Heya!”

}

]

}

Furthermore, the endpoint was changed to be “https://api.sendgrid.com/v3/mail/send”. To incorporate all these changes with the minimum number of modified lines in the codebase, it was required for that the structure change itself happens at a fairly low level. This was because there are lots of features in the server that perform a wide variety of email actions. Thus, it was clear that changing all of them will not be the most efficient thing to do. So the perfect place to implement the API changes was the function send_email() in mail.py, because all other higher-level email functions are built on top of this function. But this was not the only change, because this function itself used another function, called send_email_task() in tasks.py, specifically for sending email via Sendgrid. So, in conclusion, the header modifications were made in send_email() and payload structure as well as endpoint modifications were made within send_email_task(). This brought the server codebase back on track to send emails successfully. Finally, the key for development server was also renewed and added to its settings in the Heroku Postgres database.

Screenshots:

Screen Shot 2018-08-21 at 3.40.12 PM.png

Screen Shot 2018-08-21 at 3.40.32 PM.png

Resources

Continue Reading

Migrating Event Ratings of Open Event with Stored Procedures

Many developers know about procedural languages and have used them in some form or another, but this is really an unpopular tool, despite its power. There are many advantages (and few disadvantages) of these languages, which we will learn about soon. Having a right amount of database-stored procedure code with the help of these languages can really enhance the speed and responsiveness of an application. This article will teach us how procedural languages can be utilized in database management and how they were used recently for a bug fix in Open Event Server.

PostgreSQL, like any other powerful, relational database management system (RDBMS), provides the functionality to create and use stored procedures. Essentially, a stored procedure is database logic code which is saved on the database server. This code can be executed directly in the database, and can (and is!) often used to shift business logic from the application layer of a software to the database layer. This simple shift often has many advantages – including faster execution (as code executes at a lower stack level) and better security. When firing database queries from the application layer (i.e., the code that programmers write for storing programmable objects, performing business logic and so on), it often happens that parameters from the programming language itself are passed in to SQL, which then generates a complete SQL query. For example, here’s how a novice query might look like:

import psycopg2
conn = psycopg2.connect(dbname="oevent", user="john", password="start")
cur = conn.cursor()
name = "Sam"
cur.execute("SELECT * FROM users WHERE name='%s'" % name)  # DANGEROUS!

This is an extremely “exposed” code that can be exploited for malicious access, with a technique called SQL injection. This technique essentially “injects” malicious code via these passed parameters, like the variable name mentioned in the above code. With having stored procedures for business logic, there is no room for SQL injection. They solve this problem by writing the query beforehand, and having the parameterized data as a different entity. The pre-processed query within the corresponding stored procedure now looks like

SELECT * FROM users WHERE name=?

 

The database driver sends the name of this stored procedure (or, in standard parameterised queries, just the query text itself) and a list of parameters, as distinct separate entities in the protocol. More details on how stored procedures enhance security can be found here.

After learning so much about the advantages of stored procedures (which are enabled by procedural languages), let’s write one! Postgres supports multiple languages for writing stored procedures; here we will use PL/pgSQL, which is the most popular choice for Postgres. This procedural language, inspired (heavily) by Oracle’s PL/SQL language, looks very similar to SQL. To use this procedural language, we have to first install it. In Postgres, procedural languages are installed per-database, not server-wide. We can use the popular Postgres client psql for this purpose, or simply the createlang command on the command line:

$ createlang plpgsql yourdb

 

Now let’s create a simple procedure that prints the corresponding grades for the event ratings. In the psql shell, type the following:

CREATE OR REPLACE FUNCTION grade(rating NUMERIC) RETURNS TEXT AS
$$
BEGIN
IF rating < 1 THEN
RETURN 'D';
ELSIF rating < 2 THEN
RETURN 'C';
ELSIF rating < 3 THEN
RETURN 'B';
ELSIF rating < 4 THEN
RETURN 'A';
ELSE
RETURN 'A+';
END IF;
END;
$$ STRICT LANGUAGE plpgsql IMMUTABLE;

 

The first line defines the function signature – its name, parameters’ type and return type. The line specifies which procedural language to use. The IMMUTABLE keyword specifies that the stored procedure – this function, cannot modify the database by itself. The STRICT part says that the function should always return null when any of the arguments are null. So, the function is not executed when there are null arguments, rather a null result is assumed automatically. Finally, everything within the $$ quotes is the logic of our function, in PL/pgSQL syntax. As we can see, the syntax is almost identical to that of plain SQL.

Let us now create some dummy data to test the above stored procedure. In the same psql shell, type the following:

CREATE TABLE ratings(id INTEGER, rating NUMERIC);
INSERT INTO ratings VALUES (1, 2.3), (2, 0.5), (3, 1.7), (4, 4.5), (5, 3.3);

 

The ratings table now looks as follows:

yourdb=# SELECT * FROM ratings;
id | rating
----+--------
1 |    2.3
2 |    0.5
3 |    1.7
4 |    4.5
5 |    3.3
(5 rows)

To test our stored procedure, we fire the following query:

SELECT id, grade(rating) FROM ratings;

The result is:

id | grade
----+-------
1 | B
2 | D
3 | C
4 | A+
5 | A
(5 rows)

Thus, our stored procedure works perfectly! This shows how you can write code for your application in the database itself, the code which would otherwise be present in the application layer. For writing more complex stored procedures, you can always consult the excellent PostgreSQL documentation on this topic!

PL/pgSQL turned out to be especially useful for a recent bug patch in Open Event Server. The rating attribute of event feedbacks was initially (and incorrectly) of String type. When converting it to Float type, I had migrated the schema, but the migration of existing data was nontrivial. Since the earlier ratings were of type String, we weren’t really sure whether the values in the rating column will all contain numbers. Furthermore, we also wanted to round those numbers to the “nearest 0.5”, so that ratings like 4.4 and 2.2 get converted to more conventional values like 4.5 and 2.0 respectively. To accomplish this, I had to add a stored procedure in the related migration file. Here’s how it looked:

class ReplaceableObject(object):
    def __init__(self, name, sqltext):
        self.name = name
        self.sqltext = sqltext

update_rating_func = ReplaceableObject(
    "update_rating(rating text)",
    """
    RETURNS text AS $$
    DECLARE
        r NUMERIC;
    BEGIN
        r = cast(rating as NUMERIC);
        IF r < 0 OR r > 5 THEN
            RETURN '0';
        END IF;
        r = round(r*2) / 2;
        RETURN cast(r as VARCHAR);
    EXCEPTION WHEN invalid_text_representation THEN
        RETURN '0';
    END;
    $$
    STRICT
    LANGUAGE plpgsql IMMUTABLE;
    """)

We created a simple class ReplaceableObject so that the name of the function is, sort of, a separate attribute from its definition, as required by the migration library alembic. Once the stored procedure was defined as above, the upgrade() function of the migration file was modified accordingly:

def upgrade():
    op.create_or_replace_sp(update_rating_func)
    op.execute("UPDATE feedback SET rating=update_rating(rating)")
    op.execute("DROP FUNCTION update_rating(text)")

    op.alter_column('feedback', 'rating',
        existing_type=sa.VARCHAR(),
        type_=sa.Float(),
        existing_nullable=False,
        postgresql_using='rating::double precision')

Notice how the UPDATE query calls our stored procedure update_rating() to make existing ratings of the form we desire. Once that is done, the procedure is no longer needed, so it is DROP-ed and then, finally, the type is changed to Float. So indeed, procedural languages are used for real!

Also, what’s more exciting is that one can write Postgres stored procedures in Python itself! Python is among the four procedural languages supported by PostgreSQL, the other two being Perl and Tcl. Here’s a simple Python example of the grading code we wrote earlier:

CREATE OR REPLACE FUNCTION grade(rating NUMERIC) RETURNS TEXT AS
$$
    if rating < 1:
        return 'D'
    elif rating < 2:
        return 'C'
    elif rating < 3:
        return 'B'
    elif rating < 4:
        return 'A'
    else
        return 'A+'
$$ STRICT LANGUAGE plpython3u IMMUTABLE;

This should work identical to the PL/pgSQL equivalent code above. But don’t forget to change the command for installing the procedural language, for Python it looks like:

$ createlang plpython3u yourdb

Needless to say, the 3 here stands for Python3.

Even with all this flexibility and power, one must note that procedural languages are not designed to replace programming languages. Procedural code can easily become difficult to maintain, and one often needs superuser privileges to create stored procedures. Moreover, portability is a mess with them. That is especially true when moving from one database system (like Postgres) to another (like Microsoft SQL Server). Therefore, for optimum application performance and code maintainability, ensure that there is a healthy balance between your business logic residing in stored procedures and the application layer.

This post covered a lot of information about code residing in the database and its use cases. If you want to know more about best practices involving separation of business logic, please check out this thread on StackExchange’s software engineering website.

Resources

Continue Reading

Adding System Messages on Open Event Server

The Open Event Server enables organizers to manage events from concerts to conferences and meetups. It offers features for events with several tracks and venues. Event managers can create invitation forms for speakers and build schedules in a drag and drop interface. The event information is stored in a database. The system provides API endpoints to fetch the data, and to modify and update it.

The Open Event Server is based on JSON 1.0 Specification and hence build on top of Flask Rest Json API (for building Rest APIs) and Marshmallow (for Schema).

In this blog, we will talk about how to add API for accessing the System Messages on Open Event Server. The focus is on its Model updation and it’s Schema creation.

Model Updation

For the System Messages, we’ll make update model as follows

Now, let’s try to understand this Schema.

In this feature, we are providing Admin the rights to read email and notification formats used in Open Event application.

  1. First of all, there is the need to know that it has three columns notification_status, user_control_status and mail_status of type boolean.
  2. Next it has action attribute which is of type String.
  3. At last, we have hybrid properties email_message and notification_message which will return the format of email and notification respective to the action string.
  4. The hybrid properties depends on _email_message method and _notification_message method. These methods reads the MAILS and NOTIFS dictionaries and return there values corresponding to string of action key of corresponding record.

Schema Creation

For the System Messages, we’ll make our Schema as follows

Now, let’s try to understand this Schema.

In this feature, we are providing Admin the rights to read email and notification formats used in Open Event application.

  1. First of all, there is the need to know that it has three boolean properties notification_status, user_control_status and mail_status
  2. Next it has action attribute which is of type String and it’s value can be validated to have any one of the list provided in choices.
  3. At last, it has the String attributes email_message and notification_message which will return the action formats of email and notification concerning the action string provided.

So, we saw how System Messages Schema and Model is created / updated to allow Admin users to read it’s values.

Resources

Continue Reading

Adding Dredd Tests for Image Sizes on Open Event Flask Server

In this blog, we will talk about how to add dredd hooks for testing the API of Event Image Sizes and Speaker Image Sizes in Open Event Server. The focus is on adding the factory class and dredd hooks of these APIs using factory-boy python library and Dredd API testing framework.

Factory Creation

For the Event and Speaker Image Sizes, we’ll make our factory classes EventImageSizeFactory  and SpeakerImageSizeFactory as follows

Now, let’s try to understand this class.

In this class, we are writing the sample data two records of ImageSizes Model, these records corresponds to Event and Speaker Image Sizes.

  1. First of all, we inherit class factory.alchemy.SQLAlchemyModelFactory to build our sample data which for Image Sizes.
  2. Class Meta has model and sqlalchemy_session attributes. Model tells the factory class of to which model this factory class push the data to database and sqlalchemy_session is assigned with the current database session.
  3. Next, we add the attributes according to the model and Schema of Image Sizes.

Adding Dredd Hooks

For the ImageSizes, we’ll make our dredd hooks as follows

Now, let’s try to understand these tests.

In this tests, we check the API by matching the response after adding a record in these API to one which is present at API blueprint.

  1. First of all, we use decorator @hooks.before which means we first add a record in the database and then match the response we get from API say /v1/event-image-sizes with the response mentioned at Image Size > Event Image Size Details > Get Event Image Size Details in API blueprint.
  2. We create an instance of EventImageSizeFactory which is a record of model Image Sizes.
  3. This record is then returned as a response of API /v1/event-image-sizes and matches with the blueprint at Image Size > Event Image Size Details > Get Event Image Size Details

Similarly, we have added other dredd tests for PATCH method as well.

So, we saw how factory-boy python library and Dredd API testing framework helped us in testing the REST APIs on Open Event Server.

Resources

Continue Reading
Close Menu
%d bloggers like this: