Generate Requirement File for Python App for Meilix-Generator

Meilix-Generator is based upon Flask (a Python framework) which has several dependencies to fulfill before actually running the app properly. This article will guide you through the way I used it to automatically generate the requirement file for Meilix Generator app so that one doesn’t have to manually type all the requirements.

An app powered by Python always has several dependencies to fulfill to run the app successfully. The app root directory contains a file named as requirements.txt which contains the name of the dependency and their version. There are features ways to generate the requirement file for an app but the one which I will demonstrate is the best one. So I used this idea to generate the requirement file for webapp Meilix Generator.

Ways to get the requirement.txt

The internet has a featured way through which one has just to run a command to get a list of all the different dependencies within an app.

pip freeze > requirements.txt

This way will generate a bunch of dependencies that we not even required.

Why do we really require to generate a requirement file?

Yes, one may even ask that we can even write the dependency in the requirements.txt file. Why do we need a command to generate it?

Since because it will take care of two important things:
1. It will ensure that all the dependencies have been included, from user input one may forget to find some of the dependency and to include that.

  1. It will also take care of the Python Package Version Pinning which is really important. People use to version pinning for Python requirements as “>=” style. It’s important to follow “==” style because If we want to install the program in one year in the future, the required packages should be pinned to assure that the API changes in the installed packages do not break the program. Please read here for more info.

The way mentioned below will ensure to provide both these features.

How I generated it for Meilix Generator?

Meilix Generator run on Flask that require a requirement.txt file to fulfill the dependencies. Let’s get straight to the way to generate it for the project.

First we will simply create a requirements.in file in which we will simply mention all the dependencies in a simple way:

Flask
gunicorn
Werkzeug

Now we will use a command to latest packages:

pip install --upgrade -r requirements.in

#Note that if you would like to change the requirements, please edit the requirements.in file and run this command to update the dependencies

Then type this command to generate the requirements.txt file from requirements.in

pip-compile --output-file requirements.txt requirements.in

#fix the versions that definitely work for an eternity.
This will generate a file something as:

click==6.7                # via flask
Flask==0.12.2
gunicorn==19.7.1
itsdangerous==0.24        # via flask
Jinja2==2.9.6             # via flask
MarkupSafe==1.0           # via jinja2
Werkzeug==0.12.2          # via flask

Now you generated a perfect requirements.txt file with all the dependencies satisfied with proper python package pinning.

The meilix-generator repo which uses this:
https://github.com/fossasia/meilix-generator

Checking Whether Migrations Are Up To Date With The Sqlalchemy Models In The Open Event Server

In the Open Event Server, in the pull requests, if there is some change in the sqlalchemy model, sometimes proper migrations for the same are missed in the PR.

The first approach to check whether the migrations were up to date in the database was with the following health check function:

from subprocess import check_output
def health_check_migrations():
   """
   Checks whether database is up to date with migrations, assumes there is a single migration head
   :return:
   """
   head = check_output(["python", "manage.py", "db", "heads"]).split(" ")[0]
   
   if head == version_num:
       return True, 'database up to date with migrations'
   return False, 'database out of date with migrations'

 

In the above function, we get the head according to the migration files as following:

head = check_output(["python", "manage.py", "db", "heads"]).split(" ")[0]


The table alembic_version contains the latest alembic revision to which the database was actually upgraded. We can get this revision from the following line:

version_num = (db.session.execute('SELECT version_num from alembic_version').fetchone())['version_num']

 

Then we compare both of the given heads and return a proper tuple based on the comparison output.While this method was pretty fast, there was a drawback in this approach. If the user forgets to generate the migration files for the the changes done in the sqlalchemy model, this approach will fail to raise a failure status in the health check.

To overcome this drawback, all the sqlalchemy models were fetched automatically and simple sqlalchemy select queries were made to check whether the migrations were up to date.

Remember that a raw SQL query will not serve our purpose in this case as you’d have to specify the columns explicitly in the query. But in the case of a sqlalchemy query, it generates a SQL query based on the fields defined in the db model, so if migrations are missing to incorporate the said change proper error will be raised.

We can accomplish this from the following function:

def health_check_migrations():
   """
   Checks whether database is up to date with migrations by performing a select query on each model
   :return:
   """
   # Get all the models in the db, all models should have a explicit __tablename__
   classes, models, table_names = [], [], []
   # noinspection PyProtectedMember
   for class_ in db.Model._decl_class_registry.values():
       try:
           table_names.append(class_.__tablename__)
           classes.append(class_)
       except:
           pass
   for table in db.metadata.tables.items():
       if table[0] in table_names:
           models.append(classes[table_names.index(table[0])])

   for model in models:
       try:
           db.session.query(model).first()
       except:
           return False, '{} model out of date with migrations'.format(model)
   return True, 'database up to date with migrations'

 

In the above code, we automatically get all the models and tables present in the database. Then for each model we try a simple SELECT query which returns the first row found. If there is any error in doing so, False, ‘{} model out of date with migrations’.format(model) is returned, so as to ensure a failure status in health checks.

Related:

Implementing Health Check Endpoint in Open Event Server

A health check endpoint was required in the Open Event Server be used by Kubernetes to know when the web instance is ready to receive requests.

Following are the checks that were our primary focus for health checks:

  • Connection to the database.
  • Ensure sql-alchemy models are inline with the migrations.
  • Connection to celery workers.
  • Connection to redis instance.

Runscope/healthcheck seemed like the way to go for the same. Healthcheck wraps a Flask app object and adds a way to write simple health-check functions that can be used to monitor your application. It’s useful for asserting that your dependencies are up and running and your application can respond to HTTP requests. The Healthcheck functions are exposed via a user defined flask route so you can use an external monitoring application (monit, nagios, Runscope, etc.) to check the status and uptime of your application.

Health check endpoint was implemented at /health-check as following:

from healthcheck import HealthCheck
health = HealthCheck(current_app, "/health-check")

 

Following is the function for checking the connection to the database:

def health_check_db():
   """
   Check health status of db
   :return:
   """
   try:
       db.session.execute('SELECT 1')
       return True, 'database ok'
   except:
       sentry.captureException()
       return False, 'Error connecting to database'

 

Check functions take no arguments and should return a tuple of (bool, str). The boolean is whether or not the check passed. The message is any string or output that should be rendered for this check. Useful for error messages/debugging.

The above function executes a query on the database to check whether it is connected properly. If the query runs successfully, it returns a tuple True, ‘database ok’. sentry.captureException() makes sure that the sentry instance receives a proper exception event with all the information about the exception. If there is an error connecting to the database, the exception will be thrown. The tuple returned in this case will be return False, ‘Error connecting to database’.

Finally to add this to the endpoint:

health.add_check(health_check_db)

Following is the response for a successful health check:

{
   "status": "success",
   "timestamp": 1500915121.52474,
   "hostname": "shubham",
   "results": [
       {
           "output": "database ok",
           "checker": "health_check_db",
           "expires": 1500915148.524729,
           "passed": true,
           "timestamp": 1500915121.524729
       }
   ]
}

If the database is not connected the following error will be shown:

{
           "output": "Error connecting to database",
           "checker": "health_check_db",
           "expires": 1500965798.307425,
           "passed": false,
           "timestamp": 1500965789.307425
}

Related:

Designing a Remote Laboratory with PSLab using Python Flask Framework

In the introductory post about remote laboratories, a general set of tools to create a framework and handle its various aspects was also introduced. In this blog post, we will explore the implementation of several aspects of the backend app designed with python-flask, and the frontend based on EmberJS. A clear separation of the frontend and backend facilitates minimal disruption of either sections due to the other.

Implementing API methods in Python-Flask

In the Flask web server, page requests are handled via ‘routes’ , which are essentially URLs linked to a python function. Routes are also capable of handling payloads such as POST data, and various return types are also supported.

We shall use an example to demonstrate how a Sign-Up request sent from the sign-up form in the remote lab frontend for PSLab is handled.

@app.route('/signUp',methods=['POST'])
def signUp():
	"""Sign Up for Virtual Lab

	POST: Submit sign-up parameters. The following must be present:
	 inputName : The name of your account. does not need to be unique
	 inputEmail : e-mail ID used for login . must be unique.
	 inputPassword: password .
	Returns HTTP 404 when data does not exist.
	"""
	# read the posted values from the UI
	_name = request.form['inputName']
	_email = request.form['inputEmail']
	_password = request.form['inputPassword']

	# validate the received values
	if _name and _email and _password:
		_hashed_password = generate_password_hash(_password)
		newUser = User(_email, _name,_hashed_password)
		try:
			db.session.add(newUser)
			db.session.commit()
			return json.dumps({'status':True,'message':'User %s created successfully. e-mail:%s !'%(_name,_email)})
		except Exception as exc:
			reason = str(exc)
			return json.dumps({'status':False,'message':str(reason)})

 

In this example, the first line indicates that all URL requests made to <domain:port>/signUp will be handled by the function signUp . During development, we host the server on localhost, and use the default PORT number 8000, so sign-up forms must be submitted to 127.0.0.1:8000/signUp .

For deployment on a globally accessible server, a machine with a static IP, and a DNS record must be used. An example for such a deployment would be the heroku subdomain where pslab-remote is automatically deployed ; https://pslab-stage.herokuapp.com/signUp

A closer look at the above example will tell you that POST data can be accessed via the request.form dictionary, and that the sign-up routine requires inputName,inputEmail, and inputPassword. A password hash is generated before writing the parameters to the database.

Testing API methods using the Postman chrome extension

The route described in the above example requires form data to be submitted along with the URL, and we will use a rather handy developer tool called Postman to help us do this. In the frontend apps , AJAX methods are usually employed to do such tasks as well as handle the response from the server.

 

The above screenshot shows Postman being used to submit form data to /signUp on our API server running at localhost:8000 . The fields inputName, inputDescription, and inputPassword are also posted along with it.

In the bottom section, one can see that the server returned a positive status variable, as well as a descriptive message.

Submitting the sign up form via an Ember controller.
  • Setting up a template
    We first need to set up a template that we shall call sign-up.hbs , and add the following form to it. This form contains the details essential for signing up , and its submit action is linked to an action called `signMeUp` . This action will be defined in the controller which we shall explore shortly

<form class="form-signin" {{action "signMeUp" on="submit"}} >
        <label for="inputName" class="sr-only">Your Name</label>
        {{input value=inputName type="text" name="inputName" id="inputName" class="form-control" placeholder="name" required=true autofocus=true}}
        <label for="inputEmail" class="sr-only">Email address</label>
        {{input value=inputEmail type="email" name="inputEmail" id="inputEmail" class="form-control" placeholder="Email address" required=true autofocus=true}}
        <label for="inputPassword" class="sr-only">Password</label>
        {{input value=inputPassword type="password" name="inputPassword" id="inputPassword" class="form-control" placeholder="Password" required=true autofocus=true}}
         
        <button class="btn btn-lg btn-primary btn-block" type="submit">Sign Up</button>
</form>

 

  • Defining the controller
    The controller contains the actions and variables that the template links to. In this case, we require an action called signMeUp. The success, failure, and error handlers are hidden for clarity.

import Ember from 'ember';
export default Ember.Controller.extend({
  actions:{
    signMeUp() {
        var request = Ember.$.post("/signUp",
 this.getProperties("inputName","inputEmail","inputPassword"),this,'json');
        request.then(this.success.bind(this), this.failure.bind(this),
this.error.bind(this));
    },
  },
});

The signMeUp action submits the contents of the form to the signUp route on the API server, and the results are handled by functions called success, failure, or error depending on the type of response from the backend server.

Resources:

 

Running Dredd Hooks as a Flask App in the Open Event Server

The Open Event Server is based on the micro-framework Flask from its initial phases. After implementing API documentation, we decided to implement the Dredd testing in the Open Event API.

After isolating each request in Dredd testing, the real challenge is now to bind the database engine to the Dredd Hooks. And as we have been using Flask-SQLAlchemy db.Model Baseclass for building all the models and Flask, being a micro framework itself, came to our rescue as we could easily bind the database engine to the Flask app. Conventionally dredd hooks are written in pure Python, but we will running them as a self contained Flask app itself.

How to initialise this flask app in our dredd hooks. The Flask app can be initialised in the before_all hook easily as shown below:

def before_all(transaction):
    app = Flask(__name__)
    app.config.from_object('config.TestingConfig')

 

The database can be binded to the app as follows:

def before_all(transaction):
app = Flask(__name__)
app.config.from_object('config.TestingConfig')
db.init_app(app)
Migrate(app, db)

 

The challenge now is how to bind the application context when applying the database fixtures. In a normal Flask application this can be done as following:

with app.app_context():
#perform your operation

 

While for unit tests in python:

with app.test_request_context():
#perform tests

 

But as all the hooks are separate from each other, Dredd-hooks-python supports idea of a single stash list where you can store all the desired variables(a list or the name stash is not necessary).

The app and db can be added to stash as shown below:

@hooks.before_all
def before_all(transaction):
app = Flask(__name__)
app.config.from_object('config.TestingConfig')
db.init_app(app)
Migrate(app, db)
stash['app'] = app
stash['db'] = db

 

These variables stored in the stash can be used efficiently as below:

@hooks.before_each
def before_each(transaction):
with stash['app'].app_context():
db.engine.execute("drop schema if exists public cascade")
db.engine.execute("create schema public")
db.create_all()

 

and many other such examples.

Related Links:
1. Testing Your API Documentation With Dredd: https://matthewdaly.co.uk/blog/2016/08/08/testing-your-api-documentation-with-dredd/
2. Dredd Tutorial: https://help.apiary.io/api_101/dredd-tutorial/
3. Dredd Docs: http://dredd.readthedocs.io/

Building the Meilix Generator with Flask

Meilix Generator is a webapp which is used to trigger the Travis build of Meilix and mail the user the link of the iso. Meilix Generator webapp is based on Flask. This blog shows that how easy is to build a webapp and take the HTML files to render it into the webapp as well as to call and pass various function. Here I used Flask, the Python framework to render the HTML templates and send requests for various purposes (mentioned later in the article) without coding everything from scratch because of import facility of the Flask.

What is Flask?

Flask is a Python micro web framework based on Werkzeug, Jinja 2 template engine. It is used as the backbone of the webapp. It features us with a whole set of Python from which we can easily generate webapp. It is micro as it has no tools and no library itself. It come up with minimum requirements and one who needs can import different library and use it. And I used several import function for Meilix Generator like render_template, send_from_directory, etc.

Implementation (The use case in Meilix Generator)

First of all, the installation process: We will do the installation in a virtual environment. We prefer virtual environment to differentiate the Python working environment since few programs are there which require different Python versions to work.
Install virtual environment 

sudo pip install virtualenv

Now go to the folder (project) and activate it using

. venv/bin/activate

Now install Flask

pip install flask
Creating your project

Now it’s time to create a simple project in the directory.
Let’s use HTML as the frontend. In the folder create styles.css for styling and index.html template for the frontend of the page.We will make one app.py file which would look similar to this: 

from flask import Flask, render_template
app = Flask(__name__)
@app.route('/')
def index():
	"""Index page"""
	return render_template("index.html")
if __name__ == '__main__':
    app.run()

Flask looks for the / (root) path and here the root return the main template (index.html) which is the main function.

Compiling it to view the page:

export FLASK_DEBUG=1 FLASK_APP=app.py
flask run

You will find your page at http://127.0.0.1:5000

More options (how more it can help you)

  • Add more HTML template options and refer it in app.py
  • Easily use Github API  from a different .py file (this file should get import to app.py) to fetch data like: https://api.github.com/users/user_name : It will fetch user name, repos, followers and many more important information.

How I used this idea for FOSSASIA (Meilix Generator)

I used Flask for the backbone of project Meilix Generator. First, I used from function to import various library needed for the project and then made several functions for the same. Let’s understand the concept using few example:

from flask import Flask, render_template
@app.route('/about')
def about():
		#About page
		return render_template("about.html")

or

from flask import Flask, send_from_directory
@app.route('/uploads/<filename>')
def uploaded_file(filename):
		return send_from_directory(app.config['UPLOAD_FOLDER'],filename)

For more details file app.py can be found here of the Meilix Generator repository where we used the above idea.

Important Links and Repositories:

Mailing Attachments Using Terminal in Open Event Android

The latest version of Open Event Android App Generator, v2 lacked the feature of mailing the generated APK to the email ID that is entered at the start of the app generation process. This also included mailing the error logs in case of APK failure.

This is an important feature for app generator because the process of app generation is a time taking one. The users have to wait for the app to be generated so that they can download the generated APK. To avoid this, the generator can automatically email the APK as soon as it is generated.

I took up this issue a few days back and started working on it. I started with thinking about the ways through which it will be implemented. This required some discussions with the mentors and co-developers. We finalised on the following ways:

  • Using Sendgrid
  • Using SMTP

I will be discussing the implementation of both of them in this blog. The code for APK mailing starts with the function call Notification.send in generator.py

if completed and apk_path and not error:
   Notification.send(
       to=self.creator_email,
       subject='Your android application for %s has been generated ' % self.event_name,
       message='Hi,<br><br>'
               'Your android application for the \'%s\' event has been generated. '
               'And apk file has been attached along with this email.<br><br>'
               'Thanks,<br>'
               'Open Event App Generator' % self.event_name,
       file_attachment=apk_path,
       via_api=self.via_api
   )
else:
   Notification.send(
       to=self.creator_email,
       subject='Your android application for %s could not generated ' % self.event_name,
       message='Hi,<br><br> '
               'Your android application for the \'%s\' event could not generated. '
               'The error message has been provided below.<br><br>'
               '<code>%s</code><br><br>'
               'Thanks,<br>'
               'Open Event App Generator' % (self.event_name, str(error) if error else ''),
       file_attachment=apk_path,
       via_api=self.via_api
   )

This leads me to the class Notification.py. It has three functions:-

1. send(to, subject, message, file_attachment, via_api)
2. send_mail_via_smtp(payload):
3. send_email_via_sendgrid(payload):

As the name suggests, the first function:

send(to, subject, message, file_attachment, via_api)

mainly decides which service (out of smtp and sendgrid) should be used to send the email, on the basis of the input parameters (especially, the ‘EMAIL_SERVICE’ parameter that has to be set in config.py).
The function looks like as follows:

send(to, subject, message, file_attachment, via_api)

It is in the send() that the other two functions are called. If the email_service is smtp, it calls the Notification.send_mail_via_smtp(payload). Otherwise, the Notification.send_email_via_sendgrid(payload) is called.
The sendgrid function is pretty straightforward:

@staticmethod
def send_email_via_sendgrid(payload):

   key = current_app.config['SENDGRID_KEY']
   if not key:
       logger.info('Sendgrid key not defined')
       return
   headers = {
       "Authorization": ("Bearer " + key)
   }
   requests.post(
       "https://api.sendgrid.com/api/mail.send.json",
       data=payload,
       headers=headers
   )

It requires a personalised sendgrid key which is accessed from the config.py file. Apart from that it handles some errors by giving logs in celery tasks. The main line in the function that initiates the email is a POST request made using the python library ‘requests’. The request is made as follows:

 requests.post(
       "https://api.sendgrid.com/api/mail.send.json",
       data=payload,
       headers=headers
   )

The send_mail_via_smtp(payload): function looks for some configurations before sending the mail:

@staticmethod
def send_mail_via_smtp(payload):
   """
   Send email via SMTP
   :param config:
   :param payload:
   :return:
   """
   smtp_encryption = current_app.config['SMTP_ENCRYPTION']
   if smtp_encryption == 'tls':
       smtp_encryption = 'required'
   elif smtp_encryption == 'ssl':
       smtp_encryption = 'ssl'
   elif smtp_encryption == 'tls_optional':
       smtp_encryption = 'optional'
   else:
       smtp_encryption = 'none'
   config = {
       'host': current_app.config['SMTP_HOST'],
       'username': current_app.config['SMTP_USERNAME'],
       'password': current_app.config['SMTP_PASSWORD'],
       'encryption': smtp_encryption,
       'port': current_app.config['SMTP_PORT'],
   }
   mailer_config = {
       'transport': {
           'use': 'smtp',
           'host': config['host'],
           'username': config['username'],
           'password': config['password'],
           'tls': config['encryption'],
           'port': config['port']
       }
   }

   mailer = Mailer(mailer_config)
   mailer.start()
   message = Message(author=payload['from'], to=payload['to'])
   message.subject = payload['subject']
   message.plain = strip_tags(payload['message'])
   message.rich = payload['message']
   message.attach(payload['attachment'])
   mailer.send(message)
   mailer.stop()

It is using the Marrow Mailer Python library to email with attachments(APK). This Python library can be installed using
pip install marrow.mailer
To use Marrow Mailer you instantiate a marrow.mailer.Mailer object with the configuration, then pass Message instances to the Mailer instance’s send() method.

You can refer to the following guides for more information about sending emails through command line:
https://github.com/marrow/mailer is the official repo of Marrow Mailer repository.
https://pypi.python.org/pypi/marrow.mailer
More detailled information on sending emails using Sendgrid can be found here https://www.daveperrett.com/articles/2013/03/19/setting-up-sendmail-with-sendgrid-on-ubuntu/

Keep updating Build status in Meilix Generator

One of the problems we faced while working Meilix Generator was to provide user with the status of the custom ISO build in the Meilix Generator web app so we came up with the idea of checking the status of the link generated by the web app. If the link is available the status code would be 200 otherwise it would be 404.

We have used python script for checking the status of URL. For generating URL, we use the tag name which will be used as a variable to generate the URL of the unique event user wants the ISO for and the date will help in generation of link rest of the link remains the same.

tag = os.environ["TRAVIS_TAG"]
date = datetime.datetime.now().strftime('%Y%m%d')
url=https://github.com/xeon-zolt/meilix/releases/download/"+tag+"/meilix-zesty-"+date+"-i386.iso"

 

Now we will use urllib for monitoring the status of link.

req = Request(url)
    try:
        response = urlopen(req)
    except HTTPError as e:
        return('Building Your Iso')
    except URLError as e:
        return('We failed to reach the server.')
    else:
        return('Build Sucessful : ' + url)

 

After monitoring the status the next step was to update the status dynamically on the status page.

So we’llll use a status function in the flask app which is used by JavaScript to get status of the link after intervals of time.

Flask :

@app.route('/now')
def status_url():
    return (status())

 

Javascript:

<script type ="text/javascript">
let url ="/now"
function getstatus(url)
{
    fetch(url).then(function(response){
        return response.text()
    }).then(function(text){
        console.log("status",text)
        document.querySelector("div#status")
        .innerHTML = text
    })
    }
window.onload = function(){
    fetch(url).then(function(response){
        return response.text()
    }).then(function(text){
        console.log("status",text)
        document.querySelector("div#status")
        .innerHTML = text
    })
    window.setInterval(getstatus.bind(null,url),30*1000)
}
/*setInterval(function,interval in millsecs)*/
</script>

 

This covers various steps to prompt user whether the build is ready or not.

Resource

Flask App to Upload Wallpaper On the Server for Meilix Generator

We had a problem of getting a wallpaper from the user using Meilix Generator and use the wallpaper with the Meilix build scripts to generate the ISO. So, we were required to host the wallpaper on the server and downloaded by Travis CI during the build to include it in the ISO.

A solution is to render HTML templates and access data sent by POST using the request object from the flask. Redirect and url_for will be used to redirect the user once the upload is done and send_from_directory will help us to host the file under the /uploads that the user just uploaded which will be downloaded by the Travis for building the ISO.

We start by creating the HTML form marked with enctype=multipart/form-data.

<form action="upload" method="post" enctype="multipart/form-data">
        <input type="file" name="file"><br /><br />
        <input type="submit" value="Upload">
 </form>

 

First, we need imports of modules required. Most important is werkzeug.secure_filename().

import os
from flask import Flask, render_template, request, redirect, url_for, send_from_directory
from werkzeug import secure_file

 

Now, we’ll define where to upload and the type of file allowed for uploading. The path to upload directory on the server is defined by the extensions in app.config which is uploads/ here.

app.config['UPLOAD_FOLDER'] = 'uploads/'
app.config['ALLOWED_EXTENSIONS'] = set(['png', 'jpg', 'jpeg'])

 

This functions will check for valid extension for the wallpaper which are png, jpg and jpeg in this case defined above in app.config.

def allowed_file(filename):
    return '.' in filename and \
           filename.rsplit('.', 1)[1] in app.config['ALLOWED_EXTENSIONS']

 

After, getting the name of uploaded file from the user then using above function check if there are allowed file type and store it in a variable filename after that it move the files to the upload folder to save it.

Upload function check if the file name is safe and remove unsupported characters (line 3) after that moves it from a temporal folder to the upload folder. After moving, it renames the file as wallpaper so that the download link is same always which we have used in Meilix build script to download from server.

def upload():
    file = request.files['file']
    if file and allowed_file(file.filename):
        filename = secure_filename(file.filename)
        file.save(os.path.join(app.config['UPLOAD_FOLDER'], filename))
         os.rename(UPLOAD_FOLDER + filename, UPLOAD_FOLDER+'wallpaper')
         filename = 'wallpaper'

 

At this point, we have only uploaded the wallpaper and renamed the uploaded file to ‘wallpaper’ only. We cannot access the file outside the server it will result in 403 error so to make it available, the uploaded file need to be registered and then hosted using below code snippet.

We can also register uploaded_file as build_only rule and use the SharedDataMiddleware.

@app.route('/uploads/<filename>')
def uploaded_file(filename):
    return send_from_directory(app.config['UPLOAD_FOLDER'],filename)

The hosted wallpaper is used by Meilix in Travis CI to generate ISO using the download link which remains same for the uploaded wallpaper.

Why should we use secure secure_filename() function?

just imagine someone sends the following information as the filename to your app.

filename = "../../../../home/username/.sh"

 

If the number of ../ is correct and you would join this with your UPLOAD_FOLDER the hacker might have the ability to modify a file on the server’s filesystem that he or she should not modify.

Now, let’s look how the function works.

secure_filename('../../../../home/username/.sh')
'home_username_.sh'

Improving the uploads

We can add validation to the size of the file to be uploaded so that in case a user tries to upload a file too much big that may increase load on the server.

from flask import Flask, Request
app = Flask(__name__)
app.config['MAX_CONTENT_LENGTH'] = 16 * 1024 * 1024

Resources

Managing Related Endpoints in Permission Manager of Open Event API Server

Open Event API Server has its permission manager to manage all permission to different endpoints and some of the left gaps were filled by new helper method has_access. The next challenge for permission manager was to incorporate a feature many related endpoints points to the same resource.
Example:

  • /users-events-roles/<int:users_events_role_id>/user or
  • /event-invoices/<int:event_invoice_id>/user

Both endpoints point to Users API where they are fetching the record of a single user and for this, we apply the permission “is_user_itself”. This permission ensures that the logged in user is the same user whose record is asked through the API and for this we need the “user_id” as the “id” in the permission function, “is_user_itself”
Thus there is need to add the ability in permission manager to fetch this user_id from different models for different endpoints. For example, if we consider above endpoints then we need the ability to get user_id from UsersEventsRole and EventInvoice models and pass it to permission function so that it can use it for the check.

Adding support

To add support for multiple keys, we have to look for two things.

  • fetch_key_url
  • model

These two are key attributes to add this feature, fetch_key_url will take the comma separated list which will be matched with view_kwargs and model receives the array of the Model Classes which will be used to fetch the related records from the model
This snippet provides the main logic for this:

for index, mod in enumerate(model):
   if is_multiple(fetch_key_url):
       f_url = fetch_key_url[index]
   else:
       f_url = fetch_key_url
   try:
       data = mod.query.filter(getattr(mod, fetch_key_model) == view_kwargs[f_url]).one()
   except NoResultFound, e:
       pass
   else:
       found = True

if not found:
   return NotFoundError({'source': ''}, 'Object not found.').respond()

From the above snippet we are:

  • We iterate through the models list
  • Check if fetch_key_url has multiple keys or not
  • Get the key from fetch_key_url on the basis of multiple keys or single key in it.
  • We try to attempt to get object from model for the respective iteration
  • If there is any record/object in the database then it’s our data. Skipping further process
  • Else continue iteration till we get the object or to the end.

To use multiple mode

Instead of providing the single model to the model option of permission manager, provide an array of models. Also, it is optional to provide comma separated values to fetch_key_url
Now there can be scenario where you want to fetch resource from database model using different keys present on your view_kwargs
for example, consider these endpoints

  1. `/notifications/<notification_id>/event`
  2. `/orders/<order_id>/event`

Since they point to same resource and if you want to ensure that logged in user is organizer then you can use these two things as:

  1. fetch_key_url=”notification_id, order_id”
  2. model=[Notification, Order]

Permission manager will always match indexes in both options, the first key of fetch_key_url will be only used for the first key of the model and so on.
Also, fetch_key_url is an optional parameter and even in multiple mode you can provide a single value as well.  But if you provide multiple commas separated values make sure you provide all values such that no of values in fetch_key_url and model must be equal.

Resources