Patching an Attribute Type Across a Flask App

Recently, it was discovered by a contributor that the rating attribute for event feedbacks in Open Event was of type String. The type was incorrect, indeed. After a discussion, developers came concluded that it should be of type Float. In this post, I explain how to perform this simple migration task of changing a data type across a typical Flask app’s stack.

To begin this change, we first, we modify the database model. The model file for feedbacks (feedback.py) looks like the following:

from app.models import db


class Feedback(db.Model):
"""Feedback model class"""
__tablename__ = 'feedback'
id = db.Column(db.Integer, primary_key=True)
rating = db.Column(db.String, nullable=False)  # ←-- should be float
comment = db.Column(db.String, nullable=True)
user_id = db.Column(db.Integer,
db.ForeignKey('users.id', ondelete='CASCADE'))
event_id = db.Column(db.Integer,
db.ForeignKey('events.id', ondelete='CASCADE'))

def __init__(self, rating=None, comment=None, event_id=None,                                        user_id=None):
self.rating = rating  # ←-- cast here for safety
self.comment = comment
self.event_id = event_id
self.user_id = user_id





The change here is quite straightforward, and spans just 2 lines:

rating = db.Column(db.Float, nullable=False)

and

self.rating = float(rating)

We now perform the database migration using a couple of manage.py commands on the terminal. This file is different for different projects, but the migration commands essentially look the same. For Open Event Server, the manage.py file is at the root of the project directory (as is conventional). After cd’ing to the root, we execute the following commands:

$ python manage.py db migrate

and then

$ python manage.py db upgrade

These commands update our Open Event database so that the rating is now stored as a Float. However, if we execute these commands one after the other, we note that an exception is thrown:

sqlalchemy.exc.ProgrammingError: column "rating" cannot be cast automatically to type float
HINT:  Specify a USING expression to perform the conversion.
'ALTER TABLE feedback ALTER COLUMN rating TYPE FLOAT USING rating::double precision'

This happens because the migration code is ambiguous about what precision to use after converting to type float. It hints us to utilize the USING clause of PostgreSQL to do that. We accomplish this manually by using the psql client to connect to our database and command it the type change:

$ psql oevent
psql (10.1)
Type "help" for help.

oevent=# ALTER TABLE feedback ALTER COLUMN rating TYPE FLOAT USING rating::double precision

We now exit the psql shell and run the above migration commands again. We see that the migration commands pass successfully this time, and a migration file is generated. For our migration, the file looks like the following:

from alembic import op
import sqlalchemy as sa


# These values would be different for your migrations.
revision = '194a5a2a44ef'

down_revision = '4cac94c86047'


def upgrade():
op.alter_column('feedback', 'rating',
existing_type=sa.VARCHAR(),
type_=sa.Float(),
existing_nullable=False)


def downgrade():
op.alter_column('feedback', 'rating',
existing_type=sa.Float(),
type_=sa.VARCHAR(),
existing_nullable=False)

This is an auto-generated file (built by the database migration tool Alembic) and we need to specify the extra commands we used while migrating our database. Since we did use an extra command to specify the precision, we need to add it here. PostgreSQL USING clause can be added to alembic migration files via the postgresql_using keyword argument. Thus, the edited version of the upgrade function looks like the following:

def upgrade():
op.alter_column('feedback', 'rating',
existing_type=sa.VARCHAR(),
type_=sa.Float(),
existing_nullable=False,
postgresql_using='rating::double precision')

This completes our work on database migration. Migration files are useful for a variety of purposes – they allow us to easily get to a previous database state, or a new database state as suggested by a project collaborator. Just like git, they allow for easy version control and collaboration.

We didn’t finish this work after database migration. We also decided to impose limits on the rating value. We concluded that 0-5 would be a good range for rating. Furthermore, we also decided to round off the rating value to the “nearest 0.5”, so if the input rating is 2.3, it becomes 2.5. Also, if it is 4.1, it becomes 4.0. This was decided because such values are conventional for ratings across numerous web and mobile apps. So this will hopefully enable easier adoption for new users.

For the validation part, marshmallow came to rescue. It is a simple object serialization and deserialization tool for Python. So it basically allows to convert complex Python objects to JSON data for communicating over HTTP and vice-versa, among other things. It also facilitates pre-processing input data and therefore, allows clean validation of payloads. In our case, marshmallow was specifically used to validate the range of the rating attribute of feedbacks. The original feedbacks schema file looked like the following:

from marshmallow_jsonapi import fields
from marshmallow_jsonapi.flask import Schema, Relationship

from app.api.helpers.utilities import dasherize


class FeedbackSchema(Schema):
"""
Api schema for Feedback Model
"""
class Meta:
"""
Meta class for Feedback Api Schema
"""
type_ = 'feedback'
self_view = 'v1.feedback_detail'
self_view_kwargs = {'id': '<id>'}
inflect = dasherize

id = fields.Str(dump_only=True)
rating = fields.Str(required=True)  # ← need to validate this
comment = fields.Str(required=False)
event = Relationship(attribute='event',
self_view='v1.feedback_event',
self_view_kwargs={'id': '<id>'},
related_view='v1.event_detail',
related_view_kwargs={'feedback_id': '<id>'},
schema='EventSchemaPublic',
type_='event')




To validate the rating attribute, we use marshmallow’s Range class:

from marshmallow.validate import Range

Now we change the line

rating = fields.Str(required=True)

to

rating = fields.Float(required=True, validate=Range(min=0, max=5))

So with marshmallow, just about 2 lines of work implements rating validation for us!

After the validation part, what’s remaining is the rounding-off business logic. This is simple mathematics, and for getting to the “nearest 0.5” number, the formula goes as follows:

rating * 2 --> round off --> divide the result by 2

We will use Python’s built-in function (BIF) to accomplish this. To implement the business logic, we go back to the feedback model class and modify its constructor. Before this type change, the constructor looked like the following:

def __init__(self, rating=None, comment=None, event_id=None, user_id=None):
self.rating = rating
self.comment = comment
self.event_id = event_id
self.user_id = user_id

We change this by first converting the input rating to float, rounding it off and then finally assigning the result to feedback’s rating attribute. The new constructor is shown below:

def __init__(self, rating=None, comment=None, event_id=None, user_id=None):
rating = float(rating)
self.rating = round(rating*2, 0) / 2  # Rounds to nearest 0.5

self.comment = comment
self.event_id = event_id
self.user_id = user_id

This completes the rounding-off part and ultimately concludes rating’s type change from String to Float. We saw how a simple high-level type change requires editing code across multiple files and the use of different tools in between. In doing so, we thus also learned the utility of alembic and marshmallow in database migration and data validation, respectively.


Resources

Populating Database for different Event Types and Event Topics on Open Event Server

The Open Event Server enables organizers to manage events from concerts to conferences and meetups. It offers features for events with several tracks and venues. It uses the JSON 1.0 Specification and build on top of Flask Rest Json API (for building Rest APIs) and Marshmallow (for Schema). In this blog, we will talk about how to add API for accessing and updating the Speaker Image Size on Open Event Server. The focus is on its API creation.

In this blog, we will talk about how to populate database for different event types and event topics in the Open Event Server.

Populating the Database

Using populate_db,py for populating the database.

Now, let’s try to understand this now.

  1. First of all, we will write two functions create_event_topics and create_event_types in populate_db.py .
  2. In these function we will make a list of all the event topics and event types which we want to populate in database.
  3. We will loop through these lists to create their objects if not present.
  4. Last step is to call these functions in populate and populate_without_print functions in populate_db.py itself.

Resources

Adding System Image for Event Categories

The Open Event Server is using the JSON 1.0 Specification and build on top of Flask Rest Json API (for building Rest APIs) and Marshmallow (for Schema). In this blog, we will talk about how to add feature of System Image for Event Categories on Open Event Server. The focus is on Model updation, Schema updation and migrating the Database.

Model Updation

For adding System Image, we’ll update our Model EventTopic.

In this feature, we are providing rights to the Admin to add a system image for each Event Category so that if no image is given by a organizer of event on event creation then it will use the system image of that Event Category as event image by default.

Here we are adding a Column named system_image_url which is of type String. This value cannot be nullable and having a default value.

Migrating the Database

For the migrating the Database we will use simple commands.

This command runs migrations. If it cause problems naming Multiple Migration Head, then you need to run

This problem is caused when two developers push a migration file without merging two heads to achieve one head.

The above command will give us ids of two migration heads.

This command is merging two migration heads.

This command is upgrading the migrations.

Finally, we migrate the Database using above command.

Schema Updation

For the system image, we’ll update the Schema EventTopicSchema as follows

In this feature, to provide system image for each Event Category we’ll add a field named system_image_url in the Schema.

Here we are adding a field named system_image_url which is of marshmallow field type URL. This value cannot be none.

Validating the Event Image and using System Image by default

In this step, we’ll check if a event image is provided by organizer. If that is not provided then we’ll use system image of Event Category as Event Image.

Here, we will first take the event topic of event as added by the organizer. Then we will fetch the the database row in Event Topic model which has id == event_topic_id . Then we will return the system image url of that event topic to the event image.

So we saw how we could provide a default image for any event.

Resources

Adding Event Roles concerning a User on Open Event Server

The Open Event Server enables organizers to manage events from concerts to conferences and meetups. It offers features for events with several tracks and venues. Event managers can create invitation forms for speakers and build schedules in a drag and drop interface. The event information is stored in a database. The system provides API endpoints to fetch the data, and to modify and update it. The Open Event Server is based on JSON 1.0 Specification and hence build on top of Flask Rest Json API (for building Rest APIs) and Marshmallow (for Schema).

In this blog, we will talk about how to add different events role concerning a user on Open Event Server. The focus is on its model and Schema updation.

Model Updation

For the User Table, we’ll update our User Model as follows:

Now, let’s try to understand these hybrid properties.

In this feature, we are providing Admin the rights to see whether a user is acting as a organizer, co-organizer, track_organizer, moderator, attendee and registrar of any of the event or not. Here, _is_role method is used to check whether an user plays a event role like organizer, co-organizer, track_organizer, moderator, attendee and registrar or not. This is done by querying the record from UserEventsRole model. If the record is present then the returned value is True otherwise False.

Schema Updation

For the User Model, we’ll update our Schema as follows

Now, let’s try to understand this Schema.

Since all the properties will return either True or false so these all properties are set to Boolean in Schema. Here dump_only means, we will return this property in the Schema.

So, we saw how User Model and Schema is updated to show events role concerning a user on Open Event Server.

Resources

Adding Modules API on Open Event Server

The Open Event Server enables organizers to manage events from concerts to conferences and meet-ups. It offers features for events with several tracks and venues. Event managers can create invitation forms for speakers and build schedules in a drag and drop interface. The event information is stored in a database. The system provides API endpoints to fetch the data, and to modify and update it.

The Open Event Server is based on JSON 1.0 Specification and hence build on top of Flask Rest Json API (for building Rest APIs) and Marshmallow (for Schema).

In this blog, we will talk about how to add API for accessing the Modules on Open Event Server. The focus is on Schema creation and it’s API creation.

Schema Creation

For the ModuleSchema, we’ll make our Schema as follows

Now, let’s try to understand this Schema.

In this feature, we are providing Admin the rights to set whether Admin wants to include tickets, payment and donation in the open event application.

  1. First of all, we will provide three fields in this Schema, which are ticket_include, payment_include and donation_include.
  2. The very first attribute ticket_include should be Boolean as we want Admin to update it whether he wants to include ticketing system in the application from default one which is False.
  3. Next attribute payment_include should be Boolean as we want Admin to update it whether he wants to include payment system in the application from default one which is False.
  4. Next attribute donation_include should be Boolean as we want Admin to update it whether he wants to include donation system in the application from default one which is False.

API Creation

For the ModuleDetail, we’ll make our API as follows

Now, let’s try to understand this API.

In this API, we are providing Admin the rights to set whether Admin wants to include tickets, payment and donation in the open event application.

  1. First of all, there is the need to know that this API has two method GET and PATCH.
  2. Decorators shows us that only Admin has permissions to access PATCH method for this API i.e. only Admins can modify the modules .
  3. before_get method shows us that this API will give first record of Modules model irrespective of the id requested by user.
  4. Schema used here is default one of Modules
  5. Hence, GET Request is accessible to all the users.

So, we saw how Module Schema and API is created to allow users to get it’s values and Admin users to modify it’s values.

Resources

Adding multiple email support for users on Open Event Server

The Open Event Server enables organizers to manage events from concerts to conferences and meet-ups. It offers features for events with several tracks and venues. Event managers can create invitation forms for speakers and build schedules in a drag and drop interface. The event information is stored in a database. The system provides API endpoints to fetch the data, and to modify and update it.

The Open Event Server is based on JSON 1.0 Specification and hence build on top of Flask Rest Json API (for building Rest APIs) and Marshmallow (for Schema).

In this blog, we will talk about how to add support of multiple emails for a user in Open Event Server. The focus is on model and schema creation for this support.

Model Creation

For the UserEmail, we’ll make our model as follows

from app.models import db

class UserEmail(db.Model):
“””user email model class”””
__tablename__ = ‘user_emails’
id = db.Column(db.Integer, primary_key=True)
email = db.Column(db.String(120), unique=True, nullable=False)
verified = db.Column(db.Boolean, default=False)
user_id = db.Column(db.Integer, db.ForeignKey(‘users.id’, ondelete=’CASCADE’))
user = db.relationship(“User”, backref=”emails”, foreign_keys=[user_id])

def __init__(self, email=None, user_id=None):
self.email = email
self.user_id = user_id

def __str__(self):
return ‘User:’ + unicode(self.user_id).encode(‘utf-8’) + ‘ email: ‘ + unicode(self.email).encode(‘utf-8’)

def __unicode__(self):
return unicode(self.id)

Now, let’s try to understand the attributes of this model.

  1. id is most important Column required in every model to set it as primary key and to uniquely identify an UserEmail object.
  2. email is that attribute which is required hence should be unique and non-nullable.
  3. Verified attribute is used to check whether a email is verified or not (thus should be boolean)
  4. User_id is the attribute which specifies id of the user whose email is contained in the UserEmail object.
  5. Finally, a relationship with the user of id user_id and these emails (associated with the User.id == user_id) will be stored in the attribute emails in User Model.

Schema Creation

For the model UserEmail, we’ll make our schema UserEmailSchema as follows

from marshmallow_jsonapi import fields
from marshmallow_jsonapi.flask import Schema, Relationshipfrom app.api.helpers.utilities import dasherizeclass UserEmailSchema(Schema):
“””   API Schema for user email Model   “””class Meta:
“””  Meta class for user email API schema  “””
type_ = ‘user-emails’
self_view = ‘v1.user_emails_detail’
self_view_kwargs = {‘id’: ‘<id>’}
inflect = dasherize

id = fields.Str(dump_only=True)
email = fields.Email(allow_none=False)
user_id = fields.Integer(allow_none=False)
user = Relationship(attribute=’user’,
self_view=’v1.user_email’,
self_view_kwargs={‘id’: ‘<id>’},
related_view=’v1.user_detail’,
related_view_kwargs={‘user_id’: ‘<id>’},
schema=’UserSchema’,
type_=’user’
)

  • Marshmallow-jsonapi provides a simple way to produce JSON API-compliant data in any Python web framework.

Now, let’s try to understand the schema UserEmailSchema

  1. id : Same as in model id is used as uniquely identify an UserEmail object.
  2. email : Same as in model email is required thus allow_none is set to False.
  3. User_id : user_id is the id of user whose email is contained in a UserEmailSchema object.
  4. User : It tells whole attributes of the user to which this email belongs to.

So, we saw how to add multiple email support for users on Open Event Server. We just required to create a model and its schema to add this feature. Similarly, to add support for any database model in the project, we need to create Model and Schema with all the attributes as specified in the model too. This Schema creation is done with guidelines of JSONAPI 1.0 Specification using Marshmallow.

Resources

Permission Dependent Schema for Admin Settings in Open Event Server

For implementing the next version of the API in Open Event, the schema is a very important thing. It tells you exactly what all information you need to send in the body and how the response will look. In flask-rest-jsonapi, we usually mention a schema for an API which is then used for validating requests and sending response. Using decorators, we restrict who all can create, edit or get responses from a particular API endpoint. However, a scenario may so arise that you need to show data to users at different permissions level, but the amount of data shown significantly varies with the permission.

For example, for the settings API in our case. There are few informations like the app name, app tagline that we want to be available to users at all permission levels. However, informations such as aws secret key, or mailing secret keys or any other secret key, we want that to be available only to the admin and super admin. And the responses should be such that users at different permission level should feel that whatever information shown to them is complete and not missing.

So, what we do is we create different schemas, in our case 2 different schemas. Depending on the permission of the user, we show them a particular schema. In our case, the two schemas are SettingSchemaAdmin and SettingSchemaNonAdmin. In SettingSchemaAdmin, we have all the attributes or fields that are present and is accessible to the Admin and Super Admin. In the SettingSchemaNonAdmin however, we have only those fields and attributes that we want to show to all non admin users.

from flask_jwt import current_identity
 
class SettingDetail(ResourceDetail):
    """
    setting detail by id
    """
 
    def before_get(self, args, kwargs):
        kwargs['id'] = 1
        if current_identity.is_admin or current_identity.is_super_admin:
            self.schema = SettingSchemaAdmin
        else:
            self.schema = SettingSchemaNonAdmin

 

The above code helps us achieve this. If you have read previous blogs about the API server, you would already know that we are using JWT for authenticating our users. In this code, we are importing current_identity from flask_jwt. Current_identity, returns us an object of the User type which has properties such as is_admin, is_super_admin, etc. to help us identify the permission level of that user.
Using this object, we check whether the user who is making the request via jwt authentication is an admin or super admin, or just a normal registered user.

        if current_identity.is_admin or current_identity.is_super_admin:
            self.schema = SettingSchemaAdmin
        else:
            self.schema = SettingSchemaNonAdmin

 

So, if the current user sending the request is an admin, then we set the schema for the Resource manager class of the flask-rest-jsonapi as SettingSchemaAdmin, which we have already declared before containing all the fields, else, we set it as SettingSchemaNonAdmin which has limited number of attributes.