marshmallow

Read more about the article Adding Check-in Attributes to Tickets

Adding Check-in Attributes to Tickets

Recently, it was decided by the Open Event Orga App team that the event ticket API response from Open Event Server should have two additional attributes for specifying event check-in access. At first sight, it seemed that adding these options will only require changes in the orga app, but it turned out that the entire Ticket API from the server will need this addition. Implementing these attributes turned out to be quite straightforward. Specifically, the fields to be added were boolean is_checkin_restricted and auto_checkin_enabled. By default, checkin is not automatic and is restricted. Therefore, the default values for these fields were chosen to be True and False respectively. To add them, the ticket model file was changed first - due to the addition of these two columns: class Ticket(SoftDeletionModel): ... is_checkin_restricted = db.Column(db.Boolean) # <-- auto_checkin_enabled = db.Column(db.Boolean) # <-- ... def __init__(self, name=None, event_id=None, ... is_checkin_restricted=True, auto_checkin_enabled=False): self.name = name ... self.is_checkin_restricted = is_checkin_restricted self.auto_checkin_enabled = auto_checkin_enabled ... Since the ticket database model was updated, a migration had to be performed. Following shell commands (at the open event server project root) did the migration and database update and a migration file was then generated: $ python manage.py db migrate $ python manage.py db upgrade Here’s the generated migration file: from alembic import op import sqlalchemy as sa revision = '6440077182f0' down_revision = 'eaa029ebb260' def upgrade(): op.add_column('tickets', sa.Column('auto_checkin_enabled', sa.Boolean(), nullable=True)) op.add_column('tickets', sa.Column('is_checkin_restricted', sa.Boolean(), nullable=True)) def downgrade(): op.drop_column('tickets', 'is_checkin_restricted') op.drop_column('tickets', 'auto_checkin_enabled') The next code change was required in the ticket API schema. The change was essentially the same as the one added in the model file - just these 2 new fields were added: class TicketSchemaPublic(SoftDeletionSchema): ... id = fields.Str(dump_only=True) name = fields.Str(required=True) ... is_checkin_restricted = fields.Boolean(default=True) # <-- auto_checkin_enabled = fields.Boolean(default=False) # <-- event = Relationship(attribute='event', self_view='v1.ticket_event', self_view_kwargs={'id': '<id>'}, related_view='v1.event_detail', related_view_kwargs={'ticket_id': '<id>'}, schema='EventSchemaPublic', type_='event') ... Now all that remained were changes in the API documentation, which were made accordingly. This completed the addition of these two checkin attributes in the ticket API, and eventually made way to the orga app. And, these can be requested as usual by the front-end and user app as well. Resources and Links: Open Event Server pull request for checkin attributes Alembic documentation Marshmallow API Schema Reference

Continue ReadingAdding Check-in Attributes to Tickets
Read more about the article Patching an Attribute Type Across a Flask App

Patching an Attribute Type Across a Flask App

Recently, it was discovered by a contributor that the rating attribute for event feedbacks in Open Event was of type String. The type was incorrect, indeed. After a discussion, developers came concluded that it should be of type Float. In this post, I explain how to perform this simple migration task of changing a data type across a typical Flask app’s stack. To begin this change, we first, we modify the database model. The model file for feedbacks (feedback.py) looks like the following: from app.models import db class Feedback(db.Model): """Feedback model class""" __tablename__ = 'feedback' id = db.Column(db.Integer, primary_key=True) rating = db.Column(db.String, nullable=False) # ←-- should be float comment = db.Column(db.String, nullable=True) user_id = db.Column(db.Integer, db.ForeignKey('users.id', ondelete='CASCADE')) event_id = db.Column(db.Integer, db.ForeignKey('events.id', ondelete='CASCADE')) def __init__(self, rating=None, comment=None, event_id=None, user_id=None): self.rating = rating # ←-- cast here for safety self.comment = comment self.event_id = event_id self.user_id = user_id … … … The change here is quite straightforward, and spans just 2 lines: rating = db.Column(db.Float, nullable=False) and self.rating = float(rating) We now perform the database migration using a couple of manage.py commands on the terminal. This file is different for different projects, but the migration commands essentially look the same. For Open Event Server, the manage.py file is at the root of the project directory (as is conventional). After cd’ing to the root, we execute the following commands: $ python manage.py db migrate and then $ python manage.py db upgrade These commands update our Open Event database so that the rating is now stored as a Float. However, if we execute these commands one after the other, we note that an exception is thrown: sqlalchemy.exc.ProgrammingError: column "rating" cannot be cast automatically to type float HINT: Specify a USING expression to perform the conversion. 'ALTER TABLE feedback ALTER COLUMN rating TYPE FLOAT USING rating::double precision' This happens because the migration code is ambiguous about what precision to use after converting to type float. It hints us to utilize the USING clause of PostgreSQL to do that. We accomplish this manually by using the psql client to connect to our database and command it the type change: $ psql oevent psql (10.1) Type "help" for help. oevent=# ALTER TABLE feedback ALTER COLUMN rating TYPE FLOAT USING rating::double precision We now exit the psql shell and run the above migration commands again. We see that the migration commands pass successfully this time, and a migration file is generated. For our migration, the file looks like the following: from alembic import op import sqlalchemy as sa # These values would be different for your migrations. revision = '194a5a2a44ef' down_revision = '4cac94c86047' def upgrade(): op.alter_column('feedback', 'rating', existing_type=sa.VARCHAR(), type_=sa.Float(), existing_nullable=False) def downgrade(): op.alter_column('feedback', 'rating', existing_type=sa.Float(), type_=sa.VARCHAR(), existing_nullable=False) This is an auto-generated file (built by the database migration tool Alembic) and we need to specify the extra commands we used while migrating our database. Since we did use an extra command to specify the precision, we need to add it here. PostgreSQL USING clause can be added to alembic…

Continue ReadingPatching an Attribute Type Across a Flask App
Read more about the article Implementing Tax Endpoint in Open Event Server

Implementing Tax Endpoint in Open Event Server

The Open Event Server enables organizers to manage events from concerts to conferences and meetups. It offers features for events with several tracks and venues. The Event organizers may want to charge taxes on the event tickets. The Open Event Server has a Tax endpoint in order to support it. This blog goes over it’s implementation details in the project. Model First up, we will discuss what fields have been stored in the database for Tax endpoint. The most important fields are as follows: The tax rate charged in percentage The id for the Tax The registered company The country The address of the event organiser The additional message to be included as the invoice footer We also store a field to specify whether the tax should be included in the ticket price or not. Each Event can have only one associated Tax information. You can checkout the full model for reference here. Schema We have defined two schemas for the Tax endpoint. This is because there are a few fields which contain sensitive information and should only be shown to the event organizer or the admin itself while the others can be shown to the public. Fields like name and rate aren’t sensitive and can be disclosed to the public. They have been defined in the TaxSchemaPublic class. Sensitive information like the tax id, address, registered company have been included in the TaxSchema class which inherits from the TaxSchemaPublic class. You can checkout the full schema for reference here. Resources The endpoint supports all the CRUD operations i.e. Create, Read, Update and Delete. Create and Update The Tax entry for an Event can be created using a POST request to the /taxes endpoint. We analyze if the posted data contains a related event id or identifier which is necessary as every tax entry is supposed to be related with an event. Moreover we also check whether a tax entry already exists for the event or not since an event should have only one tax entry. An error is raised if that is not the case otherwise the tax entry is created and saved in the database. An existing entry can be updated using the same endpoint by making a PATCH request.   Read A Tax entry can be fetched using a GET request to the  /taxes/{tax_id}  endpoint with the id for the tax entry. The entry for an Event can also be fetched from /events/{event_id}/tax  endpoint. Delete An existing Tax entry can be deleted by making a DELETE request to the /taxes/{tax_id} endpoint with the id of the entry. We make sure the tax entry exists. An error is raised if that is not the case else we delete it from the database. References Marshmallow: http://marshmallow.readthedocs.io/en/latest/index.html Flask: http://flask.pocoo.org

Continue ReadingImplementing Tax Endpoint in Open Event Server
Read more about the article Adding Defaults Prior to Schema Validation Elegantly

Adding Defaults Prior to Schema Validation Elegantly

The Open Event Server offers features for events with several tracks and venues. When we were designing the models for the API, we wanted to add default values for some fields in case they aren’t provided by the client. This blog discusses how we have implemented in the project using python decorators that complies to the DRY principle and is easy to test and maintain. Problem Let’s first discuss the problem at hand in detail. We use Marshmallow extensively in our project. Marshmallow is an ORM/ODM/framework-agnostic library for converting complex data types, such as objects, to and from native python data types. We use it for Validating the input data, Deserializing the input data to app-level objects and Serializing app-level objects to primitive Python types. We can define Schema’s very easily using Marshmallow. It also provides an easy way to declare default values to the fields. Below is a sample schema declaration: class SampleSchema(Schema): """ Sample Schema declaration """ class Meta: """ Meta class for the Sample Schema """ type_ = 'sample-schema' id = fields.Str(dump_only=True) field_without_default = fields.Str(required=True) field_with_default = fields.Boolean(required=True, default=False) We have defined an id field for storing the unique ID of the Model. We have also defined two other fields. One of them named as “field_with_default” is a Boolean field and has a default value specified as False. When a client makes a POST/PATCH request to the server, we first validate the input data sent to us by the clients against the defined schema. Marshmallow also supports schema validation but it doesn’t support using default values during deserialization of the input data. It meant that whenever the input data had a missing field, the server would throw a validation error even for a field for which the default values are defined. It was clearly wrong since if the default values are defined, we would want that value to be used for the field. This defeats the entire purpose of declaring default values at the first place. So, we would ideally like the following behaviour from the Server: If the values are defined in the input data, use it during validation. If the value for a required field is not defined but default value has been defined in the Schema, then use that value. If no value has been defined for a required field and it doesn’t have any default value specified, then throw an error. Solution Marshmallow provides decorators like @pre_load and @post_load for adding pre-processing and post-processing methods. We can use them to add a method in each of the Schema classes which takes in the input data and the schema and adds default values to fields before we validate the input. The first approach which we took was to add the following method to each of the schema classes defined in the project. @pre_load def patch_defaults(schema, in_data): data = in_data.get('data') if data is None or data.get('attributes') is None: return in_data attributes = data.get('attributes') for name, field in schema.fields.items(): dasherized_name = dasherize(name) attribute = attributes.get(dasherized_name) if attribute is…

Continue ReadingAdding Defaults Prior to Schema Validation Elegantly

Adding JSON-API to Badgeyay Backend

Badgeyay has two main components, the Python-Flask backend server and the EmberJS frontend. EmberJS frontend uses ember data to save the data from the backend server api into the store of EmberJS frontend. To make the ember data frontend comply with backend api we need the backend server to send responses that comply with the standards of the JSON-API. What is JSON-API? As stated by JSONAPI.ORG "If you've ever argued with your team about the way your JSON responses should be formatted, JSON API can be your anti-bikeshedding tool." To put it up simply, JSON-API is a way of representing the JSON data that is being generated by the server backend. In this way we represent the JSON data in a particular way that follows the JSON-API convention. An example of data that follows json-api standards is given below: { "data": { "id": "1", "type": "posts", "attributes": { "title": "This is a JSON API data" }, "relationships": { "author": { "links": { "related": "/example/number" } }, "comments": { "links": { "related": "/example/number/article/" } "data": [ {"id": 5, "type": "example"}, {"id": 12, "type": "example"} ], } }, } } Adding JSON-API using Marshmallow-JSONAPI We proceeded on to adding json-api into the Python-Flask backend. Before we proceed to adding json-api, we first need to install marshmallow_jsonapi To install marshmallow_jsonapi $ ~ pip install marshmallow-jsonapi After installing marshmallow_jsonapi, we proceed onto making our first schema. A schema is a layer of abstraction that is provided over a database model that can be used to dump data from or into an object. This object can therefore be used to either store in database or to dump it to the EmberJS frontend. Let us create a schema for File. from marshmallow_jsonapi.flask import Schema from marshmallow_jsonapi import fields class FileSchema(Schema): class Meta: type_ = 'File' self_view = 'fileUploader.get_file' kwargs = {'id': '<id>'} id = fields.Str(required=True, dump_only=True) filename = fields.Str(required=True) filetype = fields.Str(required=True) user_id = fields.Relationship( self_url='/api/upload/get_file', self_url_kwargs={'file_id': '<id>'}, related_url='/user/register', related_url_kwargs={'id': '<id>'}, include_resource_linkage=True, type_='User' ) So we have successfully created a Schema for getting files. This schema has an id, filename and filetype. It also has a relationship with the User. Let us now create a route for this Schema. The below snippet of code is used to find a given file using this schema. @router.route('/get_file', methods=['GET']) def get_file(): input_data = request.args file = File().query.filter_by(filename=input_data.get('filename')).first() return jsonify(FileSchema().dump(file).data)   Now to get details of a file using our newly created route and schema all we need to do is use the following cURL command: $ ~ curl -X GET "http://localhost:5000/api/upload/get_file?filename={your_file_name}" You will get something like this as a response: { "data": { "attributes": { "filename": "13376967-8846-4c66-bcab-4a6b7d58aca7.csv", "filetype": "csv" }, "id": "967dc51b-289a-43a1-94c1-5cfce04b0fbf", "links": { "self": "/api/upload/get_file" }, "relationships": { "user_id": { "data": { "id": "J9v2LBIai1MOc8LijeLx7zWsP4I2", "type": "User" }, "links": { "related": "/user/register", "self": "/api/upload/get_file" } } }, "type": "File" }, "links": { "self": "/api/upload/get_file" } } Further Improvements After adding JSON-API standards to the backend API we can easily integrate it with the EmberJS frontend. Now we can work…

Continue ReadingAdding JSON-API to Badgeyay Backend
Read more about the article Adding multiple email support for users on Open Event Server

Adding multiple email support for users on Open Event Server

The Open Event Server enables organizers to manage events from concerts to conferences and meet-ups. It offers features for events with several tracks and venues. Event managers can create invitation forms for speakers and build schedules in a drag and drop interface. The event information is stored in a database. The system provides API endpoints to fetch the data, and to modify and update it. The Open Event Server is based on JSON 1.0 Specification and hence build on top of Flask Rest Json API (for building Rest APIs) and Marshmallow (for Schema). In this blog, we will talk about how to add support of multiple emails for a user in Open Event Server. The focus is on model and schema creation for this support. Model Creation For the UserEmail, we’ll make our model as follows from app.models import db class UserEmail(db.Model): """user email model class""" __tablename__ = 'user_emails' id = db.Column(db.Integer, primary_key=True) email = db.Column(db.String(120), unique=True, nullable=False) verified = db.Column(db.Boolean, default=False) user_id = db.Column(db.Integer, db.ForeignKey('users.id', ondelete='CASCADE')) user = db.relationship("User", backref="emails", foreign_keys=[user_id]) def __init__(self, email=None, user_id=None): self.email = email self.user_id = user_id def __str__(self): return 'User:' + unicode(self.user_id).encode('utf-8') + ' email: ' + unicode(self.email).encode('utf-8') def __unicode__(self): return unicode(self.id) Now, let’s try to understand the attributes of this model. id is most important Column required in every model to set it as primary key and to uniquely identify an UserEmail object. email is that attribute which is required hence should be unique and non-nullable. Verified attribute is used to check whether a email is verified or not (thus should be boolean) User_id is the attribute which specifies id of the user whose email is contained in the UserEmail object. Finally, a relationship with the user of id user_id and these emails (associated with the User.id == user_id) will be stored in the attribute emails in User Model. Schema Creation For the model UserEmail, we’ll make our schema UserEmailSchema as follows from marshmallow_jsonapi import fields from marshmallow_jsonapi.flask import Schema, Relationshipfrom app.api.helpers.utilities import dasherizeclass UserEmailSchema(Schema): """   API Schema for user email Model   """class Meta: """  Meta class for user email API schema  """ type_ = 'user-emails' self_view = 'v1.user_emails_detail' self_view_kwargs = {'id': '<id>'} inflect = dasherize id = fields.Str(dump_only=True) email = fields.Email(allow_none=False) user_id = fields.Integer(allow_none=False) user = Relationship(attribute='user', self_view='v1.user_email', self_view_kwargs={'id': '<id>'}, related_view='v1.user_detail', related_view_kwargs={'user_id': '<id>'}, schema='UserSchema', type_='user' ) Marshmallow-jsonapi provides a simple way to produce JSON API-compliant data in any Python web framework. Now, let’s try to understand the schema UserEmailSchema id : Same as in model id is used as uniquely identify an UserEmail object. email : Same as in model email is required thus allow_none is set to False. User_id : user_id is the id of user whose email is contained in a UserEmailSchema object. User : It tells whole attributes of the user to which this email belongs to. So, we saw how to add multiple email support for users on Open Event Server. We just required to create a model and its schema to add this feature. Similarly, to add…

Continue ReadingAdding multiple email support for users on Open Event Server
Read more about the article Using Marshmallow Fields in Open Event API Server

Using Marshmallow Fields in Open Event API Server

The nextgen Open Event API Server  provides API endpoints to fetch the data, and to modify and update it. These endpoints have been written using flask-rest-jsonapi, which is a flask extension to build APIs around the specifications provided by JSONAPI 1.0. This extension helps you, quoting from their website: flask-rest-jsonAPI’s data abstraction layer lets us expose the resources in a flexible way. This is achieved by using Marshmallow fields by marshmallow-jsonapi. This blog post explains how we use the marshmallow fields for building API endpoints in Open Event API Server. The marshmallow library is used to serialize, deserialize and validate input data. Marshmallow uses classes to define output schemas. This makes it easier to reuse and configure the and also extend the schemas. When we write the API Schema for any database model from the Open Event models, all the columns have to be added as schema fields in the API class. This is the API Server’s event schema using marshmallow fields: These are the Marshmallow Field classes for various types of data. You can pass the following parameters when creating a field object. The ones which are used in the API Server as described below. For the rest, you can read more on marshmallow docs. Let’s take a look at each of these fields. Each of the following snippets sample writing fields for API Schema. identifier = fields.Str(dump_only=True) This is a field of data-type String. dump_only :  This field will be skipped during deserialization as it is set to True here. Setting this true essentially means marking `identifier` as read-only( for HTTP API)  name = fields.Str(required=True) This again is a field of data-type String. This is a required field and a ValidationError is raised if found missing during deserialization. Taking a look at the database backend: Since this field is set to non-nullable in the database model, it is made required in API Schema.  external_event_url = fields.Url(allow_none=True) This is a field of datatype URL. Since this is not a required field, NULL values are allowed for this field in the database model. To reflect the same in the API, we have to add allow_none=True. If missing=None is unset, it defaults to false. ends_at = fields.DateTime(required=True, timezone=True) Field of datatype DateTime It is a required field for an event and the time used here is timezone aware. latitude = fields.Float(validate=lambda n: -90 <= n <= 90, allow_none=True) Field of datatype Float. In marshmallow fields, we can use validator clauses on the input value of a field using the validate: parameter. It returns a boolean, which when false raises a Validation Error. These validators are called during deserialization. is_map_shown = fields.Bool(default=False) Field of datatype boolean. Default value for the marshmallow fields can be set by defining using default: Here, is_map_shown attribute is set to false as default for an event. privacy = fields.Str(default="public") privacy is set to default “public”. When the input value for a field is missing during serialization, the default value will be used. This parameter can either be a value or a callable. As described in the examples above, you can…

Continue ReadingUsing Marshmallow Fields in Open Event API Server