Managing Related Endpoints in Permission Manager of Open Event API Server

Open Event API Server has its permission manager to manage all permission to different endpoints and some of the left gaps were filled by new helper method has_access. The next challenge for permission manager was to incorporate a feature many related endpoints points to the same resource.
Example:

  • /users-events-roles/<int:users_events_role_id>/user or
  • /event-invoices/<int:event_invoice_id>/user

Both endpoints point to Users API where they are fetching the record of a single user and for this, we apply the permission “is_user_itself”. This permission ensures that the logged in user is the same user whose record is asked through the API and for this we need the “user_id” as the “id” in the permission function, “is_user_itself”
Thus there is need to add the ability in permission manager to fetch this user_id from different models for different endpoints. For example, if we consider above endpoints then we need the ability to get user_id from UsersEventsRole and EventInvoice models and pass it to permission function so that it can use it for the check.

Adding support

To add support for multiple keys, we have to look for two things.

  • fetch_key_url
  • model

These two are key attributes to add this feature, fetch_key_url will take the comma separated list which will be matched with view_kwargs and model receives the array of the Model Classes which will be used to fetch the related records from the model
This snippet provides the main logic for this:

for index, mod in enumerate(model):
   if is_multiple(fetch_key_url):
       f_url = fetch_key_url[index]
   else:
       f_url = fetch_key_url
   try:
       data = mod.query.filter(getattr(mod, fetch_key_model) == view_kwargs[f_url]).one()
   except NoResultFound, e:
       pass
   else:
       found = True

if not found:
   return NotFoundError({'source': ''}, 'Object not found.').respond()

From the above snippet we are:

  • We iterate through the models list
  • Check if fetch_key_url has multiple keys or not
  • Get the key from fetch_key_url on the basis of multiple keys or single key in it.
  • We try to attempt to get object from model for the respective iteration
  • If there is any record/object in the database then it’s our data. Skipping further process
  • Else continue iteration till we get the object or to the end.

To use multiple mode

Instead of providing the single model to the model option of permission manager, provide an array of models. Also, it is optional to provide comma separated values to fetch_key_url
Now there can be scenario where you want to fetch resource from database model using different keys present on your view_kwargs
for example, consider these endpoints

  1. `/notifications/<notification_id>/event`
  2. `/orders/<order_id>/event`

Since they point to same resource and if you want to ensure that logged in user is organizer then you can use these two things as:

  1. fetch_key_url=”notification_id, order_id”
  2. model=[Notification, Order]

Permission manager will always match indexes in both options, the first key of fetch_key_url will be only used for the first key of the model and so on.
Also, fetch_key_url is an optional parameter and even in multiple mode you can provide a single value as well.  But if you provide multiple commas separated values make sure you provide all values such that no of values in fetch_key_url and model must be equal.

Resources

Custom Data Layer in Open Event API Server

Open Event API Server uses flask-rest-jsonapi module to implement JSON API. This module provides a good logical abstraction in the data layer.
The data layer is a CRUD interface between resource manager and data. It is a very flexible system to use any ORM or data storage. The default layer you get in flask-rest-jsonapi is the SQLAlchemy ORM Layer and API Server makes use of default alchemy layer almost everywhere except the case where I worked on email verification part.

To add support for adding user’s email verification in API Server, there was need to create an endpoint for POST /v1/users/<int:user_id>/verify
Clearly here we are working on a single resource i.e, specific user record. This requires us to use ResourceDetail and the only issue was there is no any POST method or view in ResourceDetail class. To solve this I created a custom data layer which enables me to redefine all methods and views by inheriting abstract class. A custom data layer must inherit from flask_rest_jsonapi.data_layers.base.Base.

Creating Custom Layer

To solve email verification process, a custom layer was created at app/api/data_layers/VerifyUserLayer.py

def create_object(self, data, view_kwargs):
   user = safe_query(self, User, 'id', view_kwargs['user_id'], 'user_id')
   s = get_serializer()
   try:
       data = s.loads(data['token'])
   except Exception:
       raise UnprocessableEntity({'source': 'token'}, "Invalid Token")

   if user.email == data[0]:
       user.is_verified = True
       save_to_db(user)
       return user
   else:
       raise UnprocessableEntity({'source': 'token'}, "Invalid Token")

Using custom layer in API

We can easily provide custom layer in API Resource using one of the properties of the Resource Class

data_layer = {
   'class': VerifyUserLayer,
   'session': db.session
}

This is all we have to provide in the custom layer, now all CRUD method will be directed to our custom data layer.

Solution to our issue
Setting up custom layer provides us the ability to create our custom resource methods, i.e, modifying the view for POST request and allowing us to verify the registered users in API Server.
On Setting up the data layer all I need to do is create a ResourceList with using this layer and with permissions

class VerifyUser(ResourceList):

   methods = ['POST', ]
   decorators = (jwt_required,)
   schema = VerifyUserSchema
   data_layer = {
       'class': VerifyUserLayer,
       'session': db.session
   }

This enables me to use the custom layer, VerifyUserLayer for ResourceList resource.

Resources

A guide to use Permission Manager in Open Event API Server

This article provides a simple guide to use permission manager in Open Event API Server. Permission manager is constantly being improved and new features are being added into it. To ensure that all co-developers get to know about it and make use of them, this blog posts describes every part of permission manager.

Bootstrapping

Permission manager as a part of flask-rest-jsonapi works as a decorator for different resources of the API. There are two ways to provide the permission decorator to any view

  • First one is to provide it in the list of decorators
decorators = (api.has_permission('is_coorganizer', fetch="event_id",
                                fetch_as="event_id", model=StripeAuthorization),)
    • Second way is to explicitly provide it as a decorator to any view
@api.has_permission('custom_arg', custom_kwargs='custom_kwargs')
    def get(*args, **kwargs):
        return 'Hello world !'

In the process of booting up, we first need to understand the flow of Resources in API. All resources even before doing any schema check, call the decorators. So this way you will not get any request data in the permission methods. All you will receive is a dict of the URL parameters but again it will not include the filter parameters.
Permission Manager receives five parameters as: 

def permission_manager(view, view_args, view_kwargs, *args, **kwargs):

First three are provided into it implicitly by flask-rest-jsonapi module

  • view: This is the resource’s view method which is called through the API. For example, if I go to /events then the get method of ResourceList will be called.
  • view_args: These are args associated with that view.
  • view_kwargs: These are kwargs associated with that resource view. It includes all your URL parameters as well.
  • args: These are the custom args which are provided when calling the permission manager. Here at permission manager is it expected that the first index of args will be the name of permission to check for.
  • kwargs: This is the custom dict which is provided on calling the permission manager. The main pillar of the permission manager. Described below in usage.

Using Permission Manager

Using permission manager is basically understanding the different options you can send through the kwargs so here is the list of the things you can send to permission manager
These are all described in the order of priority in permission manager

  • method (string): You can provide a string containing the methods where permission needs to be checked as comma separated values of different methods in a string.
    For example: method=”GET,POST”
  • leave_if (lambda): This receives a lambda function which should return boolean values. Based on returned value if is true then it will skip the permission check. The provided lambda function receives only parameter, “view_kwargs”
    Example use case can be the situation where you can leave the permission for any specifically related endpoint to some resource and would like to do a manual check in the method itself.
  • check (lambda): Opposite to leave_if. It receives a lambda function that will return boolean values. Based on returned value, If it is true then only it will go further and check the request for permissions else will throw forbidden error.
  • fetch (string): This is the string containing the name of the key which has to be fetched for the fetch_as key (described below). Permission manager will first look for this value in view_kwargs dict object. If it is not there then it will make the query to get one(described below at model )
  • fetch_as (string): This is the string containing the name of a key. The value of fetch key will be sent to the permission functions by this name.
  • model (string): This is one most interesting concept here. To get the value of the fetch key. Permission manager first looks into view_kwargs and if there no such value then you can still get one through the model. The model attribute here receives the class of the database model which will be used to get the value of the fetch key.
    It makes the query to get the single resource from this model and look for the value of the fetch key and then pass it to the permission functions/methods.
    The interesting part is that by default it uses <id> from view_kwargs to get the resource from the model but in any case if there is no specific ID with name <id> on the view_kwargs. You can use these two options as:
  • fetch_key_url (string): This is the name of the key whose value will be fetched from view_kwargs and will be used to match the records in database model to get the resource.
  • fetch_key_model (string): This is the name of the match column in the database model for the fetch_key_url, The value of it will be matched with the column named as the value of fetch_key_model.
    In case there is no record found in the model then permission manager will throw NotFound 404 Error.

A helper for permissions

The next big thing in permission manager is the addition of new helper function “has_access”

def has_access(access_level, **kwargs):
   if access_level in permissions:
       auth = permissions[access_level](lambda *a, **b: True, (), {}, (), **kwargs)
       if type(auth) is bool and auth is True:
           return True
   return False

This method allows you to check the permission at the mid of any method of any view and of any resource. Just provide the name of permission in the first parameter and then the additional options needed by the permission function as the kwargs values.
This does not throw any exception. Just returns the boolean value so take care of throwing any exception by yourselves.

Anything to improve on?

I will not say this exactly as the improvement but I would really like to make it more meaningful and interesting to add permission. May be something like this below:

permission = "Must be co_organizer OR track_organizer, fetch event_id as event_id, use model Event"

This clearly needs time to make it. But I see this as an interesting way to add permission. Just provide meaningful text and rest leave it to the permission manager.

Image Uploading in Open Event API Server

Open Event API Server manages image uploading in a very simple way. There are many APIs such as “Event API” in API Server provides you data pointer in request body to send the image URL. Since you can send only URLs here if you want to upload any image you can use our Image Uploading API. Now, this uploading API provides you a temporary URL of your uploaded file. This is not the permanent storage but the good thing is that developers do not have to do anything else. Just send this temporary URL to the different APIs like the event one and rest of the work is done by APIs.
API Endpoints which receives the image URLs have their simple mechanism.

  • Create a copy of an uploaded image
  • Create different sizes of the uploaded image
  • Save all images to preferred storage. The Super Admin can set this storage in admin preferences

To better understand this, consider this sample request object to create an event

{
  "data": {
    "attributes": {
      "name": "New Event",
      "starts-at": "2002-05-30T09:30:10+05:30",
      "ends-at": "2022-05-30T09:30:10+05:30",
      "email": "[email protected]",
      "timezone": "Asia/Kolkata",
      "original-image-url": "https://cdn.pixabay.com/photo/2013/11/23/16/25/birds-216412_1280.jpg"
    },
    "type": "event"
  }
}

I have provided one attribute as “original-image-url”, server will open the image and create different images of different sizes as

      "is-map-shown": false,
      "original-image-url": "http://example.com/media/events/3/original/eUpxSmdCMj/43c6d4d2-db2b-460b-b891-1ceeba792cab.jpg",
      "onsite-details": null,
      "organizer-name": null,
      "can-pay-by-stripe": false,
      "large-image-url": "http://example.com/media/events/3/large/WEV4YUJCeF/f819f1d2-29bf-4acc-9af5-8052b6ab65b3.jpg",
      "timezone": "Asia/Kolkata",
      "can-pay-onsite": false,
      "deleted-at": null,
      "ticket-url": null,
      "can-pay-by-paypal": false,
      "location-name": null,
      "is-sponsors-enabled": false,
      "is-sessions-speakers-enabled": false,
      "privacy": "public",
      "has-organizer-info": false,
      "state": "Draft",
      "latitude": null,
      "starts-at": "2002-05-30T04:00:10+00:00",
      "searchable-location-name": null,
      "is-ticketing-enabled": true,
      "can-pay-by-cheque": false,
      "description": "",
      "pentabarf-url": null,
      "xcal-url": null,
      "logo-url": null,
      "can-pay-by-bank": false,
      "is-tax-enabled": false,
      "ical-url": null,
      "name": "New Event",
      "icon-image-url": "http://example.com/media/events/3/icon/N01BcTRUN2/65f25497-a079-4515-8359-ce5212e9669f.jpg",
      "thumbnail-image-url": "http://example.com/media/events/3/thumbnail/U2ZpSU1IK2/4fa07a9a-ef72-45f8-993b-037b0ad6dd6e.jpg",

We can clearly see that server is generating three other images on permanent storage as well as creating the copy of original-image-url into permanent storage.
Since we already have our Storage class, all we need to do is to make the little bit changes in it due to the decoupling of the Open Event. Also, I had to work on these points below

  • Fix upload module, provide support to generate url of locally uploaded file based on static_domain defined in settings
  • Using PIL create a method to generate new image by converting first it to jpeg(lower size than png) and resize it according to the aspect ratio
  • Create a helper method to create different sizes
  • Store all images in preferred storage.
  • Update APIs to incorporate this feature, drop any URLs in image pointers except original_image_url

Support for generating locally uploaded file’s URL
Here I worked on adding support to check if any static_domain is set by a user and used the request.url as the fallback.

if get_settings()['static_domain']:
        return get_settings()['static_domain'] + \
            file_relative_path.replace('/static', '')
    url = urlparse(request.url)
    return url.scheme + '://' + url.host + file_relative_path

Using PIL create a method to create image

This method is created to create the image based on any size passed it to as a parameter. The important role of this is to convert the image into jpg and then resize it on the basis of size and aspect ratio provided.
Earlier, in Orga Server, we were directly using the “open” method to open Image files but since they are no longer needed to be on the local server, a user can provide the link to any direct image. To add this support, all we needed is to use StringIO to turn the read string into a file-like object

image_file = cStringIO.StringIO(urllib.urlopen(image_file).read())

Next, I have to work on clearing the temporary images from the cloud which was created using temporary APIs. I believe that will be a cakewalk for locally stored images since I already had this support in this method.

if remove_after_upload:
        os.remove(image_file)

Update APIs to incorporate this feature
Below is an example how this works in an API.

if data.get('original_image_url') and data['original_image_url'] != event.original_image_url:
            uploaded_images = create_save_image_sizes(data['original_image_url'], 'event', event.id)
            data['original_image_url'] = uploaded_images['original_image_url']
            data['large_image_url'] = uploaded_images['large_image_url']
            data['thumbnail_image_url'] = uploaded_images['thumbnail_image_url']
            data['icon_image_url'] = uploaded_images['icon_image_url']
        else:
            if data.get('large_image_url'):
                del data['large_image_url']
            if data.get('thumbnail_image_url'):
                del data['thumbnail_image_url']
            if data.get('icon_image_url'):
                del data['icon_image_url']

Here the method “create_save_image_sizes” provides the different URL of different images of different sizes and we clearly dropping any other images of different sizes is provided by the user.

General Suggestion
Sometimes when we work on such issues there are some of the things to take care of for example, if you checked the first snippet, I tried to ensure that you will get the URL although it is sure that static_domain will not be blank, because even if the user (admin) doesn’t fill that field then it will be filled by server hostname
A similar situation is the one where there is no record in Image Sizes table, may be server admin didn’t add one. In that case, it will use the standard sizes stored in the codebase to create different images of different sizes.

Resources:

Permission Manager in Open Event API Server

Open Event API Server uses different decorators to control permissions for different access levels as discussed here. Next challenging thing for permissions was reducing redundancy and ensuring permission decorators are independent of different API views. They should not look to the view for which they are checking the permission or some different logic for different views.

In API Server, we have different endpoints that leads to same Resource this way we maintain relationships between different entities but this leads to a problem where permission decorators has to work on different API endpoints that points to different or same resource and but to check a permission some attributes are required and one or more endpoints may not provide all attributes required to check a permission.

For instance, PATCH /session/id` request requires permissions of a Co-Organizer and permission decorator for this requires two things, user detail and event details. It is easy to fetch user_id from logged in user while it was challenging to get “event_id”. Therefore to solve this purpose I worked on a module named “permission_manager.py” situated at “app/api/helpers/permission_manager.py” in the codebase

Basic Idea of Permission Manager

Permission manager basically works to serve the required attributes/view_kwargs to permission decorators so that these decorators do not break

Its logic can be described as:

    1. It first sits in the middle of a request and permission decorator
    2. Evaluates the arguments passed to it and ensure the current method of the request (POST, GET, etc ) is the part of permission check or not.
    3. Uses two important things, fetch and fetch_as
      fetch => value of this argument is the URL parameter key which will be fetched from URL or the database ( if not present in URL )
      fetch_as => the value received from fetch will be sent to permission decorator by the name as the value of this option.
    4. If the fetch key is not there in URL, It uses third parameter model which is Model if the table from where this key can be fetched and then passes it to permission decorator
    5. Returns the requested view on passing access level and Forbidden error if fails

This way it ensures that if looks for the only specific type of requests allowing us to set different rules for different methods.

if 'methods' in kwargs:
        methods = kwargs['methods']

    if request.method not in methods:
        return view(*view_args, **view_kwargs)

Implementing Permission Manager

Implementing it was a simple thing,

  1. Firstly, registration of JSON API app is shifted from app/api/__init__.py to app/api/bootstrap.py so that this module can be imported anywhere
  2. Added permission manager to the app_v1 module
  3. Created permission_manager.py in app/api/helpers
  4. Added it’s usage in different APIs

An example Usage:

decorators = (api.has_permission('is_coorganizer', fetch='event_id', fetch_as="event_id", methods="POST",
                                     check=lambda a: a.get('event_id') or a.get('event_identifier')),)

Here we are checking if the request has the permission of a Co-Organizer and for this, we need to fetch event_id  from request URI. Since no model is provided here so it is required for event_id in URL this also ensures no other endpoints can leak the resource. Also here we are checking for only POST requests thus it will pass the GET requests as it is no checking.

What’s next in permission manager?

Permission has various scopes for improving, I’m still working on a module as part of permission manager which can be used directly in the middle of views and resources so that we can check for permission for specific requests in the middle of any process.

The ability to add logic so that we can leave the check on the basis of some logic may be adding some lambda attributes will work.

Resources

Decorators in Open Event API Server

One of the interesting features of Python is the decorator. Decorators dynamically alter the functionality of a function, method, or class without having to directly use subclasses or change the source code of the function being decorated.

Open Event API Server makes use of decorator in various ways. The ability to wrap a function and run the decorator(s) before executing that function solves various purpose in Python. Earlier before decoupling of Orga Server into API Server and Frontend, decorators were being used for routes, permissions, validations and more.

Now, The API Server mainly uses decorators for:

  • Permissions
  • Filtering on the basis of view_kwargs or injecting something into view_kwargs
  • Validations

We will discuss here first two because validations are simple and we are using them out of the box from marshmall-api

The second one is custom implementation made to ensure no separate generic helpers are called which can add additional database queries and call overheads in some scenarios.

Permissions Using Decorators

Flask-rest-jsonapi provides an easy way to add decorators to Resources. This is as easy as defining this into Resource class

  1. decorators = (some_decorator, )

On working to event role decorators to use here, I need to follow only these 3 rules

  • If the user is admin or super admin, he/she has full access to all event roles
  • Then check the user’s role for the given event
  • Returns the requested resource’s view if authorized unless returns Forbidden Error response.

One of the examples is:

def is_organizer(view, view_args, view_kwargs, *args, **kwargs):
  user = current_identity
 
  if user.is_staff:
      return view(*view_args, **view_kwargs)
 
  if not user.is_organizer(kwargs['event_id']):
      return ForbiddenError({'source': ''}, 'Organizer access is required').respond()
 
  return view(*view_args, **view_kwargs)

From above example, it is clear that it is following those three guidelines

Filtering on the basis of view_kwargs or injecting something into view_kwargs

This is the main point to discuss, starting from a simple scenario where we have to show different events list for different users. Before decoupling API server, we had two different routes, one served the public events listing on the basis of event identifier and other to show events to the event admins and managers, listing only their own events to their panel.

In API server there are no two different routes for this. We manage this with a single route and served both cases using the decorator. This below is the magic decorator function for this purpose

def accessible_role_based_events(view, view_args, view_kwargs, *args, **kwargs):
  if 'POST' in request.method or 'withRole' in request.args:
      _jwt_required(app.config['JWT_DEFAULT_REALM'])
      user = current_identity
      if 'GET' in request.method and user.is_staff:
          return view(*view_args, **view_kwargs)
      view_kwargs['user_id'] = user.id
  return view(*view_args, **view_kwargs)

It works simply by looking for ‘withRole’ in requests and make a decision to include user_idinto kwargs as per these rules

  1. If the request is POST then it has to be associated with some user so add the user_id
  2. If the request is GET and ‘withRole’ GET parameter is present in URL then yes add the user_id. This way user is asking to list the events in which I have some admin or manager role
  3. If the request is GET and ‘withRole’ is defined but the logged in user is admin or super_adminthen there is no need add user_id since staff can see all events in admin panel
  4. The last one is GET and no ‘withRole’ parameter is defined therefore ignores and continues the same request to list all events.

The next work is of query method of EventList Resource

if view_kwargs.get('user_id'):
          if 'GET' in request.method:
              query_ = query_.join(Event.roles).filter_by(user_id=view_kwargs['user_id']) \
                  .join(UsersEventsRoles.role).filter(Role.name != ATTENDEE)

This query joins the UsersEventsRoles model whenever user_id is defined. Thus giving role-based events only.

The next interesting part is the Implementation of permission manager to ensure permission decorators doesn’t break at any point. We will see it in next post.

References:

https://wiki.python.org/moin/PythonDecorators

Relationships and its usage in Open Event Orga Server

JSON API is a specification for writing RESTFul APIs (CRUD interfaces). This specification basically sets the standard for a client to request the resources and how a server is supposed to response minimizing the redundancy and number of requests.

If we look at the general implementation of RESTful APIs, we see that we are working on creating every endpoint manually, there are no relations. Sometimes different endpoints are being created for some slightly different business logic than other. We solve this purpose specifically the relationships using JSON API spec.

Features of JSON API

Apart from CRUD interface, JSON-API-Spec provides

  • Fetching Resources
  • Fetching Relationships
  • Inclusion of Related Resources
  • Sparse Fieldsets
  • Sorting
  • Pagination
  • Filtering

For Open Event API Server we need these below

  • Proper relationship definitions
  • Sorting
  • Filtering
  • Pagination

So JSON-API spec is a good choice for us at Orga Server since it solves our every basic need.

Overview of Changes

Firstly the main task was shifting to the library flask-rest-jsonapi because this library stands to our four needs in API. The changes included:

  • ensuring JSON-API spec in our requests and responses (although the most of the work is done by the library)
  • Reusing the current implementation of JWT authorization.
  • To locate the new API to /v1. Since Orga server is going to be API server with Open Event system following the API-centric approach, therefore, there is no need to have /api/v1
  • Now out timestamps in response and request will be timezone aware thus following ISO 8601 with timezone information (Eg. 2017-05-22T09:12:44+00:00)

Media type to use: application/vnd.api+json

A Relationship in JSON API

To begin with APIs, I started working on Sessions API of Orga server and the relation of a session with the event was represented as one of the attribute of Schema of the Session API like this below,

event = Relationship(attribute='event',
                        self_view='v1.session_event',
                        self_view_kwargs={'id': '<id>'},
                        related_view='v1.event_detail',
                        related_view_kwargs={'session_id': '<id>'},
                        schema='EventSchema',
                        type_='event')


  • attribute: name of the attribute with which this will be referenced in response API
  • self_view: A view name which represents the view of this relationship. This is a relationship endpoint of sessions API.
  • self_view_kwargs: view_kwargs for self_view, this is used to provide ID of the specific record to the relationship endpoint.
  • related_view: An endpoints to the related API/Object. Here the related object is ‘event’ so I have provided the endpoint to get the event detail.
  • related_view_kwargs: Here we can provide kwargs to the related object’s endpoint. Here we are sending the value of <Session_id> URL parameter on the related endpoint by mapping it with “id” of the current session object.
  • Schema: this is the schema of the related object. Since we have related object is event, therefore, added EventSchema of it.
  • type_: this is the type of related object which is event here.

After defining them, the magic here is no need to define and inject the relationship endpoints in the responses. We just need to add one route to v1.event_detail and we have relationship ready.

To make this work, I added these on routes file:

  • ‘/sessions/<int:session_id>/event’ to the v1.event_detail
  • api.route(SessionRelationship, ‘session_event’,
             ‘/sessions/<int:id>/relationships/event’)

And we have Relationship ready as Session -> Event in the API ready. We can use these relationships to Get the relationship Object(s), Updated them or Delete them. This helps Orga Server is a very efficient scale since many of our endpoints are related with events directly so instead of separately defining relationships we are able to do this with the help of JSON API and flask-rest-jsonapi

An Example Response

HTTP/1.1 200 OK
Content-Type: application/vnd.api+json

{
  "data": {
    "relationships": {
      "event": {
        "links": {
          "self": "/v1/speakers-calls/3/relationships/event",
          "related": "/v1/speakers-calls/3/event"
        }
      }
    },
    "attributes": {
      "announcement": "Google",
      "ends-at": "2023-05-30T09:30:10+00:00",
      "hash": null,
      "starts-at": "2022-05-30T09:30:10+00:00",
      "privacy": "public"
    },
    "type": "speakers-call",
    "id": "3",
    "links": {
      "self": "/v1/speakers-calls/3"
    }
  },
  "jsonapi": {
    "version": "1.0"
  },
  "links": {
    "self": "/v1/speakers-calls/3"
  }
}

Above example shows the relationships in the response object. We can directly check in the application using these APIs that to which type of objects this object is related with and endpoints to get related data.

Next steps in the implementation are Docs for APIs, permissions implementations to secure the endpoints and setting up unit testing of the endpoints which will be discussed in next posts.

Using HTTMock to mock Third Party APIs for Development of Open Event API server

In the process of implementing the connected social media in Open Event API server, there was a situation where we need to mock third party API services like Google OAuth, Facebook Graph API. In mocking, we try to run our tests on APIs Simulation instead of Original API such that it responds with dummy data similar to original APIs.

To implement this first we need the library support in our Orga Server to mock APIs, so the library that was used for this purpose is the httmock library for Python. One of my mentors @hongquan helped me to understand the approach that we will follow to get this implemented. So according to implementation, when we make a HTTP request to any API through tests then our implementation with httmock will be such that it

  • stands in the middle of the request,
  • Stops the request from going to the original API,
  • and returns a dummy response as if the response is from original API.

The content of this response is written by us in the test case. We have to make sure that it is same type of object as we receive from original API.

Steps to follow ( on mocking Google OAuth API )

  1. Look for response object on two requests (OAuth and profile details).
  2. Create the dummy response using the sample response object.
  3. Creating endpoints using the httpmock library.
  4. During test run, calling the specific method with HTTMock

Sample object of OAuth Response from Google is:

{
"access_token":"2YotnFZFEjr1zCsicMWpAA",
"token_type":"Bearer",
"expires_in":3600,
"refresh_token":"tGzv3JOkF0XG5Qx2TlKWIA",
"example_parameter":"example_value"
}

and from the sample object of Google Profile API we needed the link of profile for our API-server:

{'link':'http://google.com/some_id'}

 

Creating the dummy response

Creating dummy response was easy. All I had to do is provide proper header and content in response and use @urlmatch decorator

# response for getting userinfo from google

@urlmatch(netloc='https://www.googleapis.com/userinfo/v2/me')
def google_profile_mock(url, request):
   headers = {'content-type': 'application/json'}
   content = {'link':'http://google.com/some_id'}
   return response(200, content, headers, None, 5, request)

@urlmatch(netloc=r'(.*\.)?google\.com$')
def google_auth_mock(url, request):
   headers = {'content-type': 'application/json'}
   content = {
       "access_token":"2YotnFZFEjr1zCsicMWpAA",
       "token_type":"Bearer",
       "expires_in":3600,
       "refresh_token":"tGzv3JOkF0XG5Qx2TlKWIA",
       "example_parameter":"example_value"
   }
   return response(200, content, headers, None, 5, request)

 

So now we have the end points to mock the response. All we need to do is to use HTTMock inside the test case.

To use this setup all we need to do is:

with HTTMock(google_auth_mock, google_profile_mock):
                self.assertTrue('Open Event' in self.app.get('/gCallback/?state=dummy_state&code=dummy_code',
                                                          follow_redirects=True).data) 
            self.assertEqual(self.app.get('/gCallback/?state=dummy_state&code=dummy_code').status_code, 302)
            self.assertEqual(self.app.get('/gCallback/?state=dummy_state&code=dummy_code').status_code, 302)

And we were able to mock the Google APIs in our test case. Complete implementation in FOSSASIA API-Server can be seen here

 

Connecting Social Apps with Open Event Orga API Server

Orga API Server serves the organizer with many features but there was need of one feature which will allow us to provide Organizer an option to connect with their social media audience directly from API server. This will also allow the Orga Server users to share their experience for different events on their social media platforms.

For this feature, some of the social media platforms added are:

  • FaceBook
  • Twitter
  • Instagram
  • Google+

    Before connecting with these social media platforms, we have to implement and test The Auth Tool – OAuth 2.0

Without going on introductory introduction to OAuth 2.0, Let’s focus on its implementation in Orga API Server

OAuth Roles

OAuth defines four roles:

  • Resource Owner
  • Client
  • Resource Server
  • Authorization Server

Let’s look at the responsibility of these roles when you connect your social apps with API server

> Resource Owner: User/Organizer ( You )

You as User or Organizer connection your accounts with API server are the resource owners who authorize API server to access your account. During authorization, you provide us access to read your account details like Name, Email, Profile photo, etc.

> Resource / Authorization Server: Social Apps

The resource server here is your social platforms/apps where you have your account registered. These Apps provide us limited access to fetch details of your account once you authorize our application to do so. They make sure the token we provide match with the authorization provided before through your account.

> Client: Orga API Server

Orga API Server acts as the client to access your account details. Before it may do so, it must be authorized by the user, and the authorization must be validated by the API.

The process to add

A simple work plan to follow:

  1. Understanding how OAuth is implemented.
  2. Test OAuth implementation on all 4 social medias.
  3. After Necessary correction. Make sure we have all views(routes) to connect these 4 social medias.
  4. Implementing the same feature on the template file.
  5. Make sure these connect buttons are shown only when Admin has registered its client credentials in Settings.
  6. Creating a view to unlink your social media account.

Understanding how OAuth is implemented.

Current Implementation of Oauth is very simple and interesting on API server. We have Oauth helper classes which provide all necessary endpoints and different methods to get the job done.

Test OAuth implementation on all 4 social medias.

Now we can work on testing on the callbacks of all 4 social apps. We have callback defined in views/util_routes.py For this, I picked up the auth OAuth URLs and called them directly on my browsers and testing their callback. Now on callback, those methods required some change to save user data on database thus connecting their accounts with API server. This lead to changes in update_user_details and on callback methods.

def update_user_details(first_name=None,
                        last_name=None,
                        facebook_link=None,
                        twitter_link=None,
                        file_url=None,
                        instagram=None,
                        google=None):

Make sure we have all views(routes) to connect these 4 social medias

This has to be done on views/users/profile.py Addition of one method

@profile.route('/google_connect/', methods=('GET', 'POST'))
def google_connect():
        ....
        ....
    return redirect(gp_auth_url)

and testing, correction on other 3 methods

Implementing the same feature on template file.

Updating gentelella/users/settings/pages/applications.html to add changes required to add this feature. This included ability to show URLs of connected accounts and functioning connect and disconnect button

Make sure these connect buttons are shown only when Admin has registered its client credentials in Settings.

    fb = get_settings()['fb_client_id'] != None and get_settings()['fb_client_secret'] != None
        ....
        ....
        ...

The addition of such snippet provides data to the template to decide whether to show those fields or not. It will not make any sense if there is no application created to connect those accounts by Admin.

Creating a view to unlink your social media account.

utils_routes.route('/unlink-social/<social>')
def unlink_social(social):
    if login.current_user is not None and login.current_user.is_authenticated:
        ...
        ...

A method is created to unlink the connected accounts so that users can anytime disconnect their accounts from API server.

Where to connect?

Settings > Applications

 

How it Works (GIF below )

Using Cloud storage for event exports

Open-event orga server provides the ability to the organizer to create a complete export of the event they created. Currently, when an organizer triggers the export in orga server, A celery job is set to complete the export task resulting asynchronous completion of the job. Organizer gets the download button enabled once export is ready.

Till now the main issue was related to storage of those export zip files. All exported zip files were stored directly in local storage and that even not by using storage module created under orga server.

local storage path

On a mission to solve this, I made three simple steps that I followed to solve this issue.

These three steps were:

  1. Wait for shutil.make_archive to complete archive and store it in local storage.
  2. Copy the created archive to storage ( specified by user )
  3. Delete local archive created.

The easiest part here was to make these files upload to different storage ( s3, gs, local) as we already have storage helper

def upload(uploaded_file, key, **kwargs):
    """
    Upload handler
    """

The most important logic of this issue resides to this code snippet.

    dir_path = dir_path + ".zip"
 
     storage_path = UPLOAD_PATHS['exports']['zip'].format(
         event_id = event_id
     )
     uploaded_file = UploadedFile(dir_path, dir_path.rsplit('/', 1)[1])
     storage_url = upload(uploaded_file, storage_path)
 
    if get_settings()['storage_place'] != "s3" or get_settings()['storage_place'] != 'gs':
        storage_url = app.config['BASE_DIR'] + storage_url.replace("/serve_","/")
    return storage_url

From above snippet, it is clear that we are extending the process of creating the zip. Once the zip is created we will make storage path for cloud storage and upload it. Only one thing will take the time to understand here is the last second and third line of above snippet.

if get_settings()['storage_place'] != "s3" or get_settings()['storage_place'] != 'gs':
        storage_url = app.config['BASE_DIR'] + storage_url.replace("/serve_","/")

Initial the plan was simple to serve the files through “serve_static” but then the test cases were expecting a file at this location thus I had to remove “serve_” part for local storage and then it works fine on those three steps.

Next thing on this storage process need to be discussed is the feature to delete old exports. I believe one reason to keep them would be an old backup of your event will be always there with us at our cloud storage.