flask

Read more about the article Implementing Tax Endpoint in Open Event Server

Implementing Tax Endpoint in Open Event Server

The Open Event Server enables organizers to manage events from concerts to conferences and meetups. It offers features for events with several tracks and venues. The Event organizers may want to charge taxes on the event tickets. The Open Event Server has a Tax endpoint in order to support it. This blog goes over it’s implementation details in the project. Model First up, we will discuss what fields have been stored in the database for Tax endpoint. The most important fields are as follows: The tax rate charged in percentage The id for the Tax The registered company The country The address of the event organiser The additional message to be included as the invoice footer We also store a field to specify whether the tax should be included in the ticket price or not. Each Event can have only one associated Tax information. You can checkout the full model for reference here. Schema We have defined two schemas for the Tax endpoint. This is because there are a few fields which contain sensitive information and should only be shown to the event organizer or the admin itself while the others can be shown to the public. Fields like name and rate aren’t sensitive and can be disclosed to the public. They have been defined in the TaxSchemaPublic class. Sensitive information like the tax id, address, registered company have been included in the TaxSchema class which inherits from the TaxSchemaPublic class. You can checkout the full schema for reference here. Resources The endpoint supports all the CRUD operations i.e. Create, Read, Update and Delete. Create and Update The Tax entry for an Event can be created using a POST request to the /taxes endpoint. We analyze if the posted data contains a related event id or identifier which is necessary as every tax entry is supposed to be related with an event. Moreover we also check whether a tax entry already exists for the event or not since an event should have only one tax entry. An error is raised if that is not the case otherwise the tax entry is created and saved in the database. An existing entry can be updated using the same endpoint by making a PATCH request.   Read A Tax entry can be fetched using a GET request to the  /taxes/{tax_id}  endpoint with the id for the tax entry. The entry for an Event can also be fetched from /events/{event_id}/tax  endpoint. Delete An existing Tax entry can be deleted by making a DELETE request to the /taxes/{tax_id} endpoint with the id of the entry. We make sure the tax entry exists. An error is raised if that is not the case else we delete it from the database. References Marshmallow: http://marshmallow.readthedocs.io/en/latest/index.html Flask: http://flask.pocoo.org

Continue ReadingImplementing Tax Endpoint in Open Event Server
Read more about the article Update of Python Runtime in Meilix

Update of Python Runtime in Meilix

Meilix Generator is a webapp uses flask with Python, Werkzeug, and Jinja 2. It triggers Travis to release in an ISO. It is deployed on Heroku. An older python version caused the webapp to load very slowly and it also become unsupported so there was a need to update the python version. In this blog post we walk through the update of Python in the project. We can specify an explicit version of Python to be used to run your application. For example, if you require Python 2, add the following to your Pipfile: [requires] Python_version = "2.7"   Then run $ pipnv lock to generate Pipfile.lock and push to Heroku. Another way: If we are using pip, we can supply a runtime.txt file. $ cat runtime.txt python-3.6.1   Building of the webapp (Example) The webapp build in Heroku and provide us a log. The log presents the packages installed and its version. Log also shows if any newer version is present for the package. While building webapp, we get this as a log: -----> Python app detected ! The latest version of Python 3 is python-3.6.5 (you are using python-3.6.1, which is unsupported). ! We recommend upgrading by specifying the latest version (python-3.6.5).   This confirms that we need to update python version and so thus we edited the runtime.txt Now building the same webapp, we get: Python app detected -----> Found python-3.6.1, removing -----> Installing python-3.6.5 -----> Installing pip   It already using older python, so it need first to remove the older version and then it install the latest one. The same implementation can be seen in the history. Reference: Flask - The Microframework Heroku Python Runtime

Continue ReadingUpdate of Python Runtime in Meilix
Read more about the article Meilix Build Process

Meilix Build Process

Meilix is an operating system developed in FOSSASIA. It has the capability to be easily customize which makes it different from other operating systems. It is of 32 bit right now. It is all started with build.sh script. And the basic idea is: Declaring the mirror, version, and language to use. Referring to the source file Download and updating required packages Create chroot Using chroot to execute script At the very beginning, the webapp Meilix Generator trigger Travis with the input provided in the form with the event name as the tag name of the Travis build. This will help to distinguish between the releases. Then the travis script .travis.yml executes build.sh script after changing its permission. - chmod +x ./build.sh - ./build.sh |& tee log.txt   Let’s get into the build.sh script and look into it. arch=${1:-i386}   First we select the bit version of the OS which we need to build. datafiles="image-${arch}.tar.lzma sources.list"   This provides necessary data files required to build the OS. The tar.lzma contains exactly the same file which are there in an ISO of a OS. So it gives us the capability to make changes into the system files. Then it starts to download the required packages which will help during the process. Then runs a debuild script which debuilds the metapackages. chmod +x ./scripts/debuild.sh ./scripts/debuild.sh   The debuild script actually repacks the meilix-default-setting metapackages with the required settings. Meilix-default-settings metapackage is fully explained here. Debuilding process is explained here. Then Meilix goes on installing required packages and deleting the unrequired files. Meilix build script is well commented to help to understand the whole process line by line. Then Travis release a Meilix ISO in github release and mail the user with the log in the attachment. Special Features: Meilix has some special features which makes it fully customizable: Meilix-default-setting metapackages, it contains the home folder and the changes made inside it can be seen in the OS. Metapackages in the Meilix are very helpful in installing any required package. Meilix System Lock helps to clean the system and get back to a fixed time. Reference: Flask Web Forms- Generator Shell Scripting- Meilix chroot - Debian Wiki  

Continue ReadingMeilix Build Process
Read more about the article Implementation of Features in Generator UI

Implementation of Features in Generator UI

In the early stage of development, Meilix Generator only has wallpaper and event name customization. But today the webapp has bunch of customization and features list which enables an user to design its own customizable ISO. Iteration in the form Meilix Generator came across several changes in the form throughout the time. At starting we only have an email part where the ISO get mailed, a name for the event so as to distinguish the ISO image and an image upload which will be set as the default desktop wallpaper in the ISO. Then the user gets a link which get activated after 20 minutes. Till then user have to preserve the link to download the ISO. Then we introduced a new field which contains event link and this link will be set as the homepage of the browser. And we change the basic UI of the webapp. At the same we implemented SendGrid to send the user the email link in their mail. This decreases the burden of carrying the downloadable link till the ISO becomes ready. Finally today Meilix Generator looks like this. It got some more customizable fields like providing default search engine, bookmark enabling or disabling and packages to include in the ISO. It has a link on the footer from which the latest pre-build ISO can be downloaded instantly and another link which takes user to the releases page of Meilix. Reference: SendGrid Email Delivery Service SendGrid Email API

Continue ReadingImplementation of Features in Generator UI
Read more about the article Adding Defaults Prior to Schema Validation Elegantly

Adding Defaults Prior to Schema Validation Elegantly

The Open Event Server offers features for events with several tracks and venues. When we were designing the models for the API, we wanted to add default values for some fields in case they aren’t provided by the client. This blog discusses how we have implemented in the project using python decorators that complies to the DRY principle and is easy to test and maintain. Problem Let’s first discuss the problem at hand in detail. We use Marshmallow extensively in our project. Marshmallow is an ORM/ODM/framework-agnostic library for converting complex data types, such as objects, to and from native python data types. We use it for Validating the input data, Deserializing the input data to app-level objects and Serializing app-level objects to primitive Python types. We can define Schema’s very easily using Marshmallow. It also provides an easy way to declare default values to the fields. Below is a sample schema declaration: class SampleSchema(Schema): """ Sample Schema declaration """ class Meta: """ Meta class for the Sample Schema """ type_ = 'sample-schema' id = fields.Str(dump_only=True) field_without_default = fields.Str(required=True) field_with_default = fields.Boolean(required=True, default=False) We have defined an id field for storing the unique ID of the Model. We have also defined two other fields. One of them named as “field_with_default” is a Boolean field and has a default value specified as False. When a client makes a POST/PATCH request to the server, we first validate the input data sent to us by the clients against the defined schema. Marshmallow also supports schema validation but it doesn’t support using default values during deserialization of the input data. It meant that whenever the input data had a missing field, the server would throw a validation error even for a field for which the default values are defined. It was clearly wrong since if the default values are defined, we would want that value to be used for the field. This defeats the entire purpose of declaring default values at the first place. So, we would ideally like the following behaviour from the Server: If the values are defined in the input data, use it during validation. If the value for a required field is not defined but default value has been defined in the Schema, then use that value. If no value has been defined for a required field and it doesn’t have any default value specified, then throw an error. Solution Marshmallow provides decorators like @pre_load and @post_load for adding pre-processing and post-processing methods. We can use them to add a method in each of the Schema classes which takes in the input data and the schema and adds default values to fields before we validate the input. The first approach which we took was to add the following method to each of the schema classes defined in the project. @pre_load def patch_defaults(schema, in_data): data = in_data.get('data') if data is None or data.get('attributes') is None: return in_data attributes = data.get('attributes') for name, field in schema.fields.items(): dasherized_name = dasherize(name) attribute = attributes.get(dasherized_name) if attribute is…

Continue ReadingAdding Defaults Prior to Schema Validation Elegantly

Adding JSON-API to Badgeyay Backend

Badgeyay has two main components, the Python-Flask backend server and the EmberJS frontend. EmberJS frontend uses ember data to save the data from the backend server api into the store of EmberJS frontend. To make the ember data frontend comply with backend api we need the backend server to send responses that comply with the standards of the JSON-API. What is JSON-API? As stated by JSONAPI.ORG "If you've ever argued with your team about the way your JSON responses should be formatted, JSON API can be your anti-bikeshedding tool." To put it up simply, JSON-API is a way of representing the JSON data that is being generated by the server backend. In this way we represent the JSON data in a particular way that follows the JSON-API convention. An example of data that follows json-api standards is given below: { "data": { "id": "1", "type": "posts", "attributes": { "title": "This is a JSON API data" }, "relationships": { "author": { "links": { "related": "/example/number" } }, "comments": { "links": { "related": "/example/number/article/" } "data": [ {"id": 5, "type": "example"}, {"id": 12, "type": "example"} ], } }, } } Adding JSON-API using Marshmallow-JSONAPI We proceeded on to adding json-api into the Python-Flask backend. Before we proceed to adding json-api, we first need to install marshmallow_jsonapi To install marshmallow_jsonapi $ ~ pip install marshmallow-jsonapi After installing marshmallow_jsonapi, we proceed onto making our first schema. A schema is a layer of abstraction that is provided over a database model that can be used to dump data from or into an object. This object can therefore be used to either store in database or to dump it to the EmberJS frontend. Let us create a schema for File. from marshmallow_jsonapi.flask import Schema from marshmallow_jsonapi import fields class FileSchema(Schema): class Meta: type_ = 'File' self_view = 'fileUploader.get_file' kwargs = {'id': '<id>'} id = fields.Str(required=True, dump_only=True) filename = fields.Str(required=True) filetype = fields.Str(required=True) user_id = fields.Relationship( self_url='/api/upload/get_file', self_url_kwargs={'file_id': '<id>'}, related_url='/user/register', related_url_kwargs={'id': '<id>'}, include_resource_linkage=True, type_='User' ) So we have successfully created a Schema for getting files. This schema has an id, filename and filetype. It also has a relationship with the User. Let us now create a route for this Schema. The below snippet of code is used to find a given file using this schema. @router.route('/get_file', methods=['GET']) def get_file(): input_data = request.args file = File().query.filter_by(filename=input_data.get('filename')).first() return jsonify(FileSchema().dump(file).data)   Now to get details of a file using our newly created route and schema all we need to do is use the following cURL command: $ ~ curl -X GET "http://localhost:5000/api/upload/get_file?filename={your_file_name}" You will get something like this as a response: { "data": { "attributes": { "filename": "13376967-8846-4c66-bcab-4a6b7d58aca7.csv", "filetype": "csv" }, "id": "967dc51b-289a-43a1-94c1-5cfce04b0fbf", "links": { "self": "/api/upload/get_file" }, "relationships": { "user_id": { "data": { "id": "J9v2LBIai1MOc8LijeLx7zWsP4I2", "type": "User" }, "links": { "related": "/user/register", "self": "/api/upload/get_file" } } }, "type": "File" }, "links": { "self": "/api/upload/get_file" } } Further Improvements After adding JSON-API standards to the backend API we can easily integrate it with the EmberJS frontend. Now we can work…

Continue ReadingAdding JSON-API to Badgeyay Backend
Read more about the article Open Event Server – Pages API

Open Event Server – Pages API

This article illustrates how the Pages API has been designed and implemented on the server side, i.e., FOSSASIA‘s Open Event Server. Pages endpoint is used to create static pages such as “About Page” or any other page that doesn’t need to be updated frequently and only a specific content is to be shown. Parameters name - This stores the name of the page. Type - String Required - Yes title - This stores the title of the page. Type - String Required - No url - This stores the url of the page. Type - String Required - Yes description - This stores the description of the page. Type - String Required - Yes language - This stores the language of the page. Type - String Required - No index - This stores the position of the page. Type - Integer Required - No Default - 0 place - Location where the page will be placed. Type - String Required - No Accepted Values - ‘footer’ and ‘event’ These are the allowed parameters for the endpoint. Model Lets see how we model this API. The ORM looks like this : __tablename__ = 'pages' id = db.Column(db.Integer, primary_key=True) name = db.Column(db.String, nullable=False) title = db.Column(db.String) url = db.Column(db.String, nullable=False) description = db.Column(db.String) place = db.Column(db.String) language = db.Column(db.String) index = db.Column(db.Integer, default=0) As you can see, we created a table called “pages”. This table has 8 columns, 7 of which are the parameters that I have mentioned above. The column “id” is an Integer column and is the primary key column. This will help to differentiate between the various entries in the table. The visualisation for this table looks as follows : API We support the following operations: GET all the pages in the database POST create a new page GET details of a single page as per id PATCH a single page by id DELETE a single page by id To implement this we first add the routes in our python file as follows : api.route(PageList, 'page_list', '/pages') api.route(PageDetail, 'page_detail', '/pages/<int:id>') Then we define these classes to handle the requests. The first route looks as follows: class PageList(ResourceList):   """   List and create page   """   decorators = (api.has_permission('is_admin', methods="POST"),)   schema = PageSchema   data_layer = {'session': db.session,                 'model': Page} As can be seen above, this request requires the user to be an admin. It uses the Page model described above and handles a POST request. The second route is: class PageDetail(ResourceDetail):   """   Page detail by id   """   schema = PageSchema   decorators = (api.has_permission('is_admin', methods="PATCH,DELETE"),)   data_layer = {'session': db.session,                 'model': Page} This route also requires the user to be an admin. It uses the Page model and handles PATCH, DELETE requests. To summarise our APIs are: GET /v1/pages{?sort,filter} POST /v1/pages{?sort,filter} GET /v1/pages/{page_id} PATCH /v1/pages/{page_id} DELETE /v1/pages/{page_id} References Flask-Marshmallow SQLAlchemy Open Event Server Flask

Continue ReadingOpen Event Server – Pages API

Generating Badges from Badgeyay API

Badgeyay is a badge generator and its main functionality is generating badges. Since the beginning of GSoC 2018 period, Badgeyay is under refactoring and remodeling process. We have introduced many APIs to make sure that Badgeyay works. Now, the badge generator has an endpoint to generate badges for your events/meetups How to create badges? Creating badges using the newly formed API is simpler than before. All you need to do is pass some basic details of the image you want, the data you want, the size and the color of font etc to the API and woosh! Within a blink of your eye the badges are generated. Backend requires some data fields to generate badges { "csv" : "a731h-jk12n-bbau2-saj2-nxg31.csv", "image" : "p2ja7-gna398-c23ba-naj31a.png", "text-color" : "#ffffff" } “csv” is the filename of the csv that the user uploads and get back as a result, “image” is the image name that user gets after a successful upload to the respective APIs, “text-color” is the color of the text that the user wants on the badges. Output of the API { "output" :  "path-to-the-pdf-of-the-badge-generated", . . } What is happening behind the scene? Once the user sends the data to the API, the required route is triggered and the data is checked,If the data is not present an error response is sent back to the user so as to inform them about the misplacement or improper format of data. import os from flask import Blueprint, jsonify, request from flask import current_app as app # from api.helpers.verifyToken import loginRequired from api.utils.response import Response from api.utils.svg_to_png import SVG2PNG from api.utils.merge_badges import MergeBadges router = Blueprint('generateBadges', __name__) @router.route('/generate_badges', methods=['POST']) def generateBadges(): try: data = request.get_json() except Exception as e: return jsonify( Response(401).exceptWithMessage(str(e),'Could not find any JSON')) if not data.get('csv'): return jsonify( Response(401).generateMessage('No CSV filename found')) if not data.get('image'): return jsonify(Response(401).generateMessage('No Image filename found')) csv_name = data.get('csv') image_name = data.get('image') text_color = data.get('text-color') or '#ffffff' svg2png = SVG2PNG() svg2png.do_text_fill('static/badges/8BadgesOnA3.svg', text_color) merge_badges = MergeBadges(image_name, csv_name) merge_badges.merge_pdfs() output = os.path.join(app.config.get('BASE_DIR'), 'static', 'temporary', image_name) return jsonify( Response(200).generateMessage(str(output)))   After the data is received, we send it to MergeBadges which internally calls the GenerateBadges class which creates the badges. Brief explanation of the Badge Generation Process: - Gather data from the user- Fill the SVG for badges with the text color - Load the image from uploads directory - Generate badges for every individual - Create PDFs for individual Badges - Merge those PDFs to provide an all-badges pdf to the user   And this is how we generated badges for the user using the Badgeyay Backend API. How is this effective? We are making sure that the user chooses the image and csv that he/she has uploaded only, In this way we maintain a proper workflow, we also manage these badges into the database and hence using the filenames helps a lot.It does not involve sending huge files and a lot of data like we had in the previous API. Earlier, we used to send the image and the csv…

Continue ReadingGenerating Badges from Badgeyay API

File and Image Upload API in Badgeyay

Badgeyay has seen many changes in the recent past during its refactoring. It started off with backend and we have now transition to remodeling backend as well. The backend transition is working perfectly. We have established sufficient APIs so far to get it working. Some of the most important APIs that we created are Image Upload API File Upload API Why do we need APIs? We need APIs so that the frontend written in Ember JS can coordinate with the backend written in Python Flask with the database being PostgreSQL. Creating the APIs Creating these APIs is easy and straightforward. The following APIs are written in Python Flask with a backend database support of PostgreSQL. Image Upload API The image upload API considers that the frontend is sending the Image as a base64 encoded string and the backend is supposed to accept this string and convert this string into an image and save it onto the server. We proceed by creating a file named fileUploader.py and code the following API. First of all, we need to declare the imports from flask import Blueprint, request, jsonify from api.utils.response import Response from api.helpers.verifyToken import loginRequired from api.helpers.uploads import saveToImage, saveToCSV Now, let’s create a route for image upload. router = Blueprint('fileUploader', __name__) @router.route('/image', methods=['POST']) @loginRequired def uploadImage(): try: image = request.json['data'] except Exception as e: return jsonify( Response(400).exceptWithMessage( str(e), 'No Image is specified')) extension = request.json['extension'] try: imageName = saveToImage(imageFile=image, extension=extension) except Exception as e: return jsonify( Response(400).exceptWithMessage( str(e), 'Image could not be uploaded')) return jsonify( Response(200).generateMessage({ 'message': 'Image Uploaded Successfully', 'unique_id': imageName})) We are using the saveToImage function to actually save the image to the backend server. The function definition of saveToImage function is given below. def generateFileName(): return str(uuid.uuid4())def saveToImage(imageFile=None, extension='.png'): imageName = generateFileName() + extension imageDirectory = os.path.join(app.config.get('BASE_DIR'), 'static', 'uploads', 'image')if not os.path.isdir(imageDirectory): os.makedirs(imageDirectory)imagePath = os.path.join(imageDirectory, imageName) image = open(imagePath, "wb") image.write(imageFile.decode('base64')) image.close() return imageName Similarly, we are using file upload route to upload files to backend server. The route for uploading files along with its helper function saveToCSV is given below. def saveToCSV(csvFile=None, extension='.csv'): csvName = generateFileName() + extension csvDirectory = os.path.join(app.config.get('BASE_DIR'), 'static', 'uploads', 'csv')if not os.path.isdir(csvDirectory): os.makedirs(csvDirectory)csvPath = os.path.join(csvDirectory, csvName) csvFile.save(csvPath)return csvName @router.route('/file', methods=['POST']) @loginRequired def fileUpload(): if 'file' not in request.files: return jsonify( Response(401).generateMessage( 'No file is specified'))file = request.files['file'] try: csvName = saveToCSV(csvFile=file, extension='.csv') except Exception as e: return jsonify( Response(400).exceptWithMessage( str(e), 'CSV File could not be uploaded'))return jsonify( Response(200).generateMessage({ 'message': 'CSV Uploaded successfully', 'unique_id': csvName})) What happens to the uploaded files? The uploaded files gets saved into their respective directories, i.e. static/uploads/csv for CSV files and static/uploads/images for Image uploads. The developer can view them from their respective folders. The static folder has been added to .gitignore  so that it does not gets uploaded to github repository. Everything has been taken care of with immense accuracy and proper error handling. Further Improvements Further improvements in Badgeyay includes adding separate database models, work on adding a beautiful frontend and to add proper routes for completing…

Continue ReadingFile and Image Upload API in Badgeyay
Read more about the article Database Listener for User Centric Events

Database Listener for User Centric Events

Badgeyay is an open-source utility developed by FOSSASIA to generate badges for conferences and events. The project is separated into two components to ease maintainability. First is the frontend part which is in ember and second part is backend which is in Flask. The choice of database to support backend is PostgreSQL. Now comes the problem, whenever a user is registered in the database, he should receive  a verification mail, that he is successfully registered on the platform. For this case we have to listen to the database events on User model. This issue has greater extendibility than only sending greeting or verification mail to the user. We can extend this to trigger services that are dependent on user registration, like subscribing the user to some set of services based on the plan he opted while registration and many more. These type of issues cannot be handled by normal relationship with tables and other entities, there has to be logic in place to support such functionalities. So the challenges for tackling the problem are as follows: Listen to the insert_action in User model Extracting the details necessary for the logic Execute particular logic Procedure Attaching insert_action listener to the User model. This function will get triggered whenever an entity is saved in the User model. <!-- HTML generated using hilite.me -->@db.event.listens_for(User, "after_insert") def logic(mapper, connection, target): { ...... } When the function gets triggered, extract the details of the saved user that is necessary for the logic. As currently we are sending greeting mail to the user,we only need the email of the user. Target is the actual saved user passed as argument to the listening function from the library. <!-- HTML generated using hilite.me -->msg = {} msg['subject'] = "Welcome to Badgeyay" msg['receipent'] = target.email msg['body'] = "It's good to have you onboard with Badgeyay. Welcome to " \ "FOSSASIA Family." sendMail(msg) Now the details are passed to sendMail() function for sending mail which uses flask-mail library to send mail to the recipient. def sendMail(message): if message and message.receipent: try: msg = Message( subject=message.subject, sender=app.config['MAIL_USERNAME'], Response(200).generateMessage( recipients=[message.receipent], body=message.body) Mail(app).send(msg) except Exception as e: return jsonify( Response(500).exceptWithMessage( str(e), 'Unable to send the mail')) return jsonify( Response(200).generateMessage( 'Mail Sent')) else: return jsonify( Response(403).generateMessage( 'No data received')) 'No data received')) This will send mail to the user who has been registered to the application. Similarly we can use separate logics according to the need of the application.   The Pull Request for the above functionality is at this Link Topics Involved Working on the issue involve following topics: Configuring mail service to allow insecure apps access. Sending mail from the flask-mail to end user Attaching listener to listen for database change Extraction of data from saved object in database sqlalchemy. Resources Sending Mails Programmatically -  Link Flask Mail Documentation - Link Listening to database events - Link Enabling access to GMAIL to send mails to recipient - Link

Continue ReadingDatabase Listener for User Centric Events