Open Event Server – Pages API

This article illustrates how the Pages API has been designed and implemented on the server side, i.e., FOSSASIA‘s Open Event Server. Pages endpoint is used to create static pages such as “About Page” or any other page that doesn’t need to be updated frequently and only a specific content is to be shown.

Parameters

  1. name – This stores the name of the page.
      1. Type – String
      2. Required – Yes
  2. title – This stores the title of the page.
      1. Type – String
      2. Required – No
  3. url – This stores the url of the page.
      1. Type – String
      2. Required – Yes
  4. description – This stores the description of the page.
      1. Type – String
      2. Required – Yes
  5. language – This stores the language of the page.
      1. Type – String
      2. Required – No
  6. index – This stores the position of the page.
      1. Type – Integer
      2. Required – No
      3. Default – 0
  7. place – Location where the page will be placed.
      1. Type – String
      2. Required – No
      3. Accepted Values – ‘footer’ and ‘event’

These are the allowed parameters for the endpoint.

Model

Lets see how we model this API. The ORM looks like this :

__tablename__ = 'pages'
id = db.Column(db.Integer, primary_key=True)
name = db.Column(db.String, nullable=False)
title = db.Column(db.String)
url = db.Column(db.String, nullable=False)
description = db.Column(db.String)
place = db.Column(db.String)
language = db.Column(db.String)
index = db.Column(db.Integer, default=0)

As you can see, we created a table called “pages”. This table has 8 columns, 7 of which are the parameters that I have mentioned above. The column “id” is an Integer column and is the primary key column. This will help to differentiate between the various entries in the table.

The visualisation for this table looks as follows :

API

We support the following operations:

  1. GET all the pages in the database
  2. POST create a new page
  3. GET details of a single page as per id
  4. PATCH a single page by id
  5. DELETE a single page by id

To implement this we first add the routes in our python file as follows :

api.route(PageList, 'page_list', '/pages')
api.route(PageDetail, 'page_detail', '/pages/<int:id>')

Then we define these classes to handle the requests. The first route looks as follows:

class PageList(ResourceList):
   """
   List and create page
   """
   decorators = (api.has_permission('is_admin', methods="POST"),)
   schema = PageSchema
   data_layer = {'session': db.session,
                 'model': Page}

As can be seen above, this request requires the user to be an admin. It uses the Page model described above and handles a POST request.

The second route is:

class PageDetail(ResourceDetail):
   """
   Page detail by id
   """
   schema = PageSchema
   decorators = (api.has_permission('is_admin', methods="PATCH,DELETE"),)
   data_layer = {'session': db.session,
                 'model': Page}

This route also requires the user to be an admin. It uses the Page model and handles PATCH, DELETE requests.

To summarise our APIs are:

GET

/v1/pages{?sort,filter}

POST

/v1/pages{?sort,filter}

GET

/v1/pages/{page_id}

PATCH

/v1/pages/{page_id}

DELETE

/v1/pages/{page_id}

References

Open Event Frontend – Updating Ember Models Table from V1 to V2

FOSSASIA‘s Open Event Frontend uses the Ember Models Table for rendering all its tables. This provides features like easy sorting, pagination etc. Another major feature is that it can be modified to meet our styling needs. As we use Semantic UI for styling, we added the required CSS classes to our table.

In version 1 this was done by overriding the classes, as shown below :

const defaultMessages = {
  searchLabel            : 'Search:',
  searchPlaceholder      : 'Search',


  ..... more to follow 
};

const defaultIcons = {
  sortAsc         : 'caret down icon',
  sortDesc        : 'caret up icon',
  columnVisible   : 'checkmark box icon',
  
  ..... more to follow  
};

const defaultCssClasses = {
  outerTableWrapper              : 'ui ui-table',
  innerTableWrapper              : 'ui segment column sixteen wide inner-table-wrapper',
  table                          : 'ui tablet stackable very basic table',
  globalFilterWrapper            : 'ui row',

 ... more to follow
};

const assign = Object.assign || assign;

export default TableComponent.extend({
  layout,

  _setupMessages: observer('customMessages', function() {
    const customIcons = getWithDefault(this, 'customMessages', {});
    let newMessages = {};
    assign(newMessages, defaultMessages, customIcons);
    set(this, 'messages', O.create(newMessages));
  }),

  _setupIcons() {
    const customIcons = getWithDefault(this, 'customIcons', {});
    let newIcons = {};
    assign(newIcons, defaultIcons, customIcons);
    set(this, 'icons', O.create(newIcons));
  },

  _setupClasses() {
    const customClasses = getWithDefault(this, 'customClasses', {});
    let newClasses = {};
    assign(newClasses, defaultCssClasses, customClasses);
    set(this, 'classes', O.create(newClasses));
  },

  simplePaginationTemplate: 'components/ui-table/simple-pagination',

  ........
});

And was used in the template as follows:

<div class="{{classes.outerTableWrapper}}">
  <div class="{{classes.globalFilterDropdownWrapper}}">

But in version 2, some major changes were introduced as follows:

  1. All partials inside a models-table were replaced with components
  2. models-table can now be used with block content
  3. New themes mechanism introduced for styling

Here, I will talk about how the theming mechanism has been changed. As I mentioned above, in version 1 we used custom classes and icons. In version 2 the idea itself has changed. A new type called Theme was added. It provides four themes out of the box – SemanticUI, Bootstrap4, Bootstrap3, Default.

We can create our custom theme based on any of the predefined themes. To suit our requirements we decided to modify the SemanticUI theme. We created a separate file to keep our custom theme so that code remains clean and short.

import Default from 'ember-models-table/themes/semanticui';

export default Default.extend({
 components: {
   'pagination-simple'    : 'components/ui-table/simple-pagination',
   'numericPagination'    : 'components/ui-table/numeric-pagination',
   .....  
 },

 classes: {
   outerTableWrapper              : 'ui ui-table',
   innerTableWrapper              : 'ui segment column sixteen wide inner-table-wrapper',
   .....
 },

 icons: {
   sortAsc         : 'caret down icon',
   sortDesc        : 'caret up icon',
   ......
 },

 messages: {
   searchLabel            : 'Search:',
   .....
 }
});

So a theme mostly consists of four main parts:

  • Components
  • Classes
  • Icons
  • Messages

The last three are same as customClasses and customIcons and customMessages in version 1. Components is the map for components used internally in the models-table. In case you need to use a custom component, that can be done as follows:

Make a new JavaScript file and provide its path in your theme file.

import DefaultDropdown from '../../columns-dropdown';
import layout from 'your layout file path';
export default DefaultDropdown.extend({
  layout
});

Now just create the theme file object and pass it to themeInstance in the ui-table file (can also be passed in the template and the controller, but this has to be done for each table individually).

import TableComponent from 'ember-models-table/components/models-table';
import layout from 'open-event-frontend/templates/components/ui-table';
import Semantic from 'open-event-frontend/themes/semantic';

export default TableComponent.extend({
 layout,

 themeInstance: Semantic.create()
});

Hence, version 2 introduces many new styling options and requires some refactoring for those who were using version 1. It is totally worth it though considering how easy and well managed it is now.

References

Open Event Server – Change a Column from NULL to NOT NULL

FOSSASIA‘s Open Event Server uses alembic migration files to handle all database operations and updating. Whenever the database is changed a corresponding migration python script is made so that the database will migrate accordingly for other developers as well. But often we forget that the automatically generated script usually just add/deletes columns or alters the column properties. It does not handle the migration of existing data in that column. This can lead to huge data loss or error in migration as well.

For example :

def upgrade():
    # ### commands auto generated by Alembic - please adjust! ###
    op.alter_column('ticket_holders', 'lastname',
                    existing_type=sa.VARCHAR(),
                    nullable=False)
    # ### end Alembic commands ###

Here, the goal was to change the column “ticket_holders” from nullable to not nullable. The script that alembic autogenerated just uses op.alter_column().

It does not count for the already existing data. So, if the column has any entries which are null, this migration will lead to an error saying that the column contains null entries and hence cannot be “NOT NULL”.

How to Handle This?

Before altering the column definition we can follow the following steps :

  1. Look for all the null entries in the column
  2. Give some arbitrary default value to those
  3. Now we can safely alter the column definition

Let’s see how we can achieve this. For connecting with the database we will use SQLAlchemy. First, we get a reference to the table and the corresponding column that we wish to alter.

ticket_holders_table = sa.sql.table('ticket_holders',
                                        sa.Column('lastname', sa.VARCHAR()))

 

Since we need the “last_name” column from the table “ticket_holders”, we specify it in the method argument.

Now, we will give an arbitrary default value to all the originally null entries in the column. In this case, I chose to use a space character.

op.execute(ticket_holders_table.update()
               .where(ticket_holders_table.c.lastname.is_(None))
               .values({'lastname': op.inline_literal(' ')}))

op.execute() can execute direct SQL commands as well but we chose to go with SQLAlchemy which builds an optimal SQL command from our modular input. One such example of a complex SQL command being directly executed is :

op.execute('INSERT INTO event_types(name, slug) SELECT DISTINCT event_type_id, lower(replace(regexp_replace(event_type_id, \'& |,\', \'\', \'g\'), \' \', \'-\')) FROM events where not exists (SELECT 1 FROM event_types where event_types.name=events.event_type_id) and event_type_id is not null;'))

Now that we have handled all the null data, it is safe to alter the column definition. So we proceed to execute the final statement –

op.alter_column('ticket_holders', 'lastname',
                    existing_type=sa.VARCHAR(),
                    nullable=False)

Now the entire migration script will run without any error. The final outcome would be –

  1. All the null “last_name” entries would be replaced by a space character
  2. The “last_name” column would now be a NOT NULL column.

References