Adding JSON-API to Badgeyay Backend

Badgeyay has two main components, the Python-Flask backend server and the EmberJS frontend.

EmberJS frontend uses ember data to save the data from the backend server api into the store of EmberJS frontend. To make the ember data frontend comply with backend api we need the backend server to send responses that comply with the standards of the JSON-API.

What is JSON-API?

As stated by JSONAPI.ORG

"If you've ever argued with your team about the way your JSON responses should be formatted, JSON API can be your anti-bikeshedding tool."

To put it up simply, JSON-API is a way of representing the JSON data that is being generated by the server backend. In this way we represent the JSON data in a particular way that follows the JSON-API convention. An example of data that follows json-api standards is given below:

{
"data": {
"id": "1",
"type": "posts",
"attributes": {
"title": "This is a JSON API data"
},
"relationships": {
"author": {
"links": {
"related": "/example/number"
}
},
"comments": {
"links": {
"related": "/example/number/article/"
}
"data": [
{"id": 5, "type": "example"},
{"id": 12, "type": "example"}
],
}
},
}
}

Adding JSON-API using Marshmallow-JSONAPI

We proceeded on to adding json-api into the Python-Flask backend. Before we proceed to adding json-api, we first need to install marshmallow_jsonapi

To install marshmallow_jsonapi

$ ~ pip install marshmallow-jsonapi

After installing marshmallow_jsonapi, we proceed onto making our first schema.

A schema is a layer of abstraction that is provided over a database model that can be used to dump data from or into an object. This object can therefore be used to either store in database or to dump it to the EmberJS frontend. Let us create a schema for File.

from marshmallow_jsonapi.flask import Schema
from marshmallow_jsonapi import fields


class FileSchema(Schema):
class Meta:
type_ = 'File'
self_view = 'fileUploader.get_file'
kwargs = {'id': '<id>'}

id = fields.Str(required=True, dump_only=True)
filename = fields.Str(required=True)
filetype = fields.Str(required=True)
user_id = fields.Relationship(
self_url='/api/upload/get_file',
self_url_kwargs={'file_id': '<id>'},
related_url='/user/register',
related_url_kwargs={'id': '<id>'},
include_resource_linkage=True,
type_='User'
)

So we have successfully created a Schema for getting files. This schema has an id, filename and filetype. It also has a relationship with the User.

Let us now create a route for this Schema. The below snippet of code is used to find a given file using this schema.

@router.route('/get_file', methods=['GET'])
def get_file():
input_data = request.args
file = File().query.filter_by(filename=input_data.get('filename')).first()
return jsonify(FileSchema().dump(file).data)

 

Now to get details of a file using our newly created route and schema all we need to do is use the following cURL command:

$ ~ curl -X GET "http://localhost:5000/api/upload/get_file?filename={your_file_name}"

You will get something like this as a response:

{
"data": {
"attributes": {
"filename": "13376967-8846-4c66-bcab-4a6b7d58aca7.csv",
"filetype": "csv"
},
"id": "967dc51b-289a-43a1-94c1-5cfce04b0fbf",
"links": {
"self": "/api/upload/get_file"
},
"relationships": {
"user_id": {
"data": {
"id": "J9v2LBIai1MOc8LijeLx7zWsP4I2",
"type": "User"
},
"links": {
"related": "/user/register",
"self": "/api/upload/get_file"
}
}
},
"type": "File"
},
"links": {
"self": "/api/upload/get_file"
}
}

Further Improvements

After adding JSON-API standards to the backend API we can easily integrate it with the EmberJS frontend. Now we can work on adding more schemas as a method of layers of abstraction so that the backend can serve more functionalities and the data can be consumed by the frontend as well.

Resources

Generating Badges from Badgeyay API

Badgeyay is a badge generator and its main functionality is generating badges. Since the beginning of GSoC 2018 period, Badgeyay is under refactoring and remodeling process. We have introduced many APIs to make sure that Badgeyay works. Now, the badge generator has an endpoint to generate badges for your events/meetups

How to create badges?

Creating badges using the newly formed API is simpler than before. All you need to do is pass some basic details of the image you want, the data you want, the size and the color of font etc to the API and woosh! Within a blink of your eye the badges are generated.

Backend requires some data fields to generate badges

{
"csv" : "a731h-jk12n-bbau2-saj2-nxg31.csv",
"image" : "p2ja7-gna398-c23ba-naj31a.png",
"text-color" : "#ffffff"
}

“csv” is the filename of the csv that the user uploads and get back as a result, “image” is the image name that user gets after a successful upload to the respective APIs, “text-color” is the color of the text that the user wants on the badges.

Output of the API

{
"output" :  "path-to-the-pdf-of-the-badge-generated",
.
.
}

What is happening behind the scene?

Once the user sends the data to the API, the required route is triggered and the data is checked,If the data is not present an error response is sent back to the user so as to inform them about the misplacement or improper format of data.

import os
from flask import Blueprint, jsonify, request
from flask import current_app as app
# from api.helpers.verifyToken import loginRequired
from api.utils.response import Response
from api.utils.svg_to_png import SVG2PNG
from api.utils.merge_badges import MergeBadges


router = Blueprint('generateBadges', __name__)


@router.route('/generate_badges', methods=['POST'])
def generateBadges():
try:
data = request.get_json()
except Exception as e:
return jsonify(
Response(401).exceptWithMessage(str(e),'Could not find any JSON'))

if not data.get('csv'):
return jsonify(
Response(401).generateMessage('No CSV filename found'))
if not data.get('image'):
return jsonify(Response(401).generateMessage('No Image filename found'))
csv_name = data.get('csv')
image_name = data.get('image')
text_color = data.get('text-color') or '#ffffff'
svg2png = SVG2PNG()
svg2png.do_text_fill('static/badges/8BadgesOnA3.svg', text_color)
merge_badges = MergeBadges(image_name, csv_name)
merge_badges.merge_pdfs()

output = os.path.join(app.config.get('BASE_DIR'), 'static', 'temporary', image_name)
return jsonify(
Response(200).generateMessage(str(output)))

 

After the data is received, we send it to MergeBadges which internally calls the GenerateBadges class which creates the badges.

Brief explanation of the Badge Generation Process:
- Gather data from the user- Fill the SVG for badges with the text color

- Load the image from uploads directory
- Generate badges for every individual
- Create PDFs for individual Badges
- Merge those PDFs to provide an all-badges pdf to the user

 

And this is how we generated badges for the user using the Badgeyay Backend API.

How is this effective?

We are making sure that the user chooses the image and csv that he/she has uploaded only,

In this way we maintain a proper workflow, we also manage these badges into the database and hence using the filenames helps a lot.It does not involve sending huge files and a lot of data like we had in the previous API.

Earlier, we used to send the image and the csv altogether that caused a serious mismanagement of the project. In this case we are accepting the CSVs and the Images on different API routes and then using the specific image and csv to make badges. We can now more easily relate to the files associated with each and every badge and henceforth we can easily manage them in the database.

Further Improvements

We will work on adding security to the route so that not anyone can create badges. We also need to integrate database into badges generated service so that we can maintain the badges that the user has generated.

Resources

File and Image Upload API in Badgeyay

Badgeyay has seen many changes in the recent past during its refactoring. It started off with backend and we have now transition to remodeling backend as well.

The backend transition is working perfectly. We have established sufficient APIs so far to get it working.

Some of the most important APIs that we created are

  • Image Upload API
  • File Upload API

Why do we need APIs?

We need APIs so that the frontend written in Ember JS can coordinate with the backend written in Python Flask with the database being PostgreSQL.

Creating the APIs

Creating these APIs is easy and straightforward. The following APIs are written in Python Flask with a backend database support of PostgreSQL.

Image Upload API

The image upload API considers that the frontend is sending the Image as a base64 encoded string and the backend is supposed to accept this string and convert this string into an image and save it onto the server.

We proceed by creating a file named fileUploader.py and code the following API.

First of all, we need to declare the imports

from flask import Blueprint, request, jsonify
from api.utils.response import Response
from api.helpers.verifyToken import loginRequired
from api.helpers.uploads import saveToImage, saveToCSV

Now, let’s create a route for image upload.

router = Blueprint('fileUploader', __name__)

@router.route('/image', methods=['POST'])
@loginRequired
def uploadImage():
try:
image = request.json['data']
except Exception as e:
return jsonify(
Response(400).exceptWithMessage(
str(e),
'No Image is specified'))

extension = request.json['extension']
try:
imageName = saveToImage(imageFile=image, extension=extension)
except Exception as e:
return jsonify(
Response(400).exceptWithMessage(
str(e),
'Image could not be uploaded'))

return jsonify(
Response(200).generateMessage({
'message': 'Image Uploaded Successfully',
'unique_id': imageName}))

We are using the saveToImage function to actually save the image to the backend server.

The function definition of saveToImage function is given below.

def generateFileName():
return str(uuid.uuid4())def saveToImage(imageFile=None, extension='.png'):
imageName = generateFileName() + extension
imageDirectory = os.path.join(app.config.get('BASE_DIR'), 'static', 'uploads', 'image')if not os.path.isdir(imageDirectory):
os.makedirs(imageDirectory)imagePath = os.path.join(imageDirectory, imageName)
image = open(imagePath, "wb")
image.write(imageFile.decode('base64'))
image.close()

return imageName

Similarly, we are using file upload route to upload files to backend server.

The route for uploading files along with its helper function saveToCSV is given below.

def saveToCSV(csvFile=None, extension='.csv'):
csvName = generateFileName() + extension
csvDirectory = os.path.join(app.config.get('BASE_DIR'), 'static', 'uploads', 'csv')if not os.path.isdir(csvDirectory):
os.makedirs(csvDirectory)csvPath = os.path.join(csvDirectory, csvName)
csvFile.save(csvPath)return csvName
@router.route('/file', methods=['POST'])
@loginRequired
def fileUpload():
if 'file' not in request.files:
return jsonify(
Response(401).generateMessage(
'No file is specified'))file = request.files['file']
try:
csvName = saveToCSV(csvFile=file, extension='.csv')
except Exception as e:
return jsonify(
Response(400).exceptWithMessage(
str(e),
'CSV File could not be uploaded'))return jsonify(
Response(200).generateMessage({
'message': 'CSV Uploaded successfully',
'unique_id': csvName}))

What happens to the uploaded files?

The uploaded files gets saved into their respective directories, i.e. static/uploads/csv for CSV files and static/uploads/images for Image uploads.

The developer can view them from their respective folders. The static folder has been added to .gitignore  so that it does not gets uploaded to github repository.

Everything has been taken care of with immense accuracy and proper error handling.

Further Improvements

Further improvements in Badgeyay includes adding separate database models, work on adding a beautiful frontend and to add proper routes for completing the backend.

Resources

Refactoring and Remodeling Badgeyay API

When we build a full scale production application, we make sure that everything is modeled correctly and accordingly to the need of the code. The code must be properly maintained as well as designed in such a way that it is less prone to errors and bugs.

Badgeyay is also targeting to be a full production application, and in order to achieve it we first need to re-factor the code and model it using a strong yet maintainable structure.

What is the current state of Badgeyay?

Currently Badgeyay is divided into two sub folders.

\badgeyay
    \frontend
    \backend
    .
    .

It is backed by two folders, viz backend and frontend. The ‘backend’ folder handles the API that the service is currently running. The ‘frontend’ folder houses the Ember based frontend logic of the application.

Improvements to Badgeyay Backend

We have worked on improving Backend for Badgeyay. Instead of traditional methods, i.e. current method, of API development; We employ a far better approach of using Flask Blueprint as a method of refactoring the API.

The new backend API resides inside the following structure.

\badgeyay
    \backend
        \blueprint
            \api

The API folder currently holds the new API being formatted from scratch using

  • Flask Blueprint
  • Flask Utilities like jsonify, response etc

The new structure of Badgeyay Backend will follow the following structure

api
    \config
    \controllers
    \helpers
    \models
    \utils
    db.py
    run.py

The folders and their use cases are given below

  • \config
    • Contain all the configuration files
    • Configurations about URLs, PostgreSQL etc
  • \controllers
    • This will contain the controllers for our API
    • Controllers will be the house to our routes for APIs
  • \helpers
    • Helpers folder will contain the files directly related to API
  • \models
    • Models folder contains the Schemas for PostgreSQL
    • Classes like User etc will be stored in here
  • \utils
    • Utils will contain the helper functions or classes
    • This classes or functions are not directly connected to the APIs
  • db.py
    • Main python file for Flask SQLAlchemy
  • run.py
    • This is the main entry point.
    • Running this file will run the entire Flask Blueprint API

How does it help?

  • It helps in making the backend more solid.
  • It helps in easy understanding of application with maintained workflow.
  • Since we will be adding a variety of features during Google Summer of Code 2018 therefore we need to have a well structured API with well defined paths for every file being used inside it.
  • It will help in easy maintaining for any maintainer on this project.
  • Development of the API will be faster in this way, since everything is divided into sub parts therefore many people can work on many different possibilities on the same time.

Further Improvements

Since this structure has been setup correctly in Badgeyay now, so we can work on adding separate routes and different functionalities can be added simultaneously.

It ensures faster development of the project.

Resources