Postgresql

Read more about the article Implementing Database Migrations to Badgeyay

Implementing Database Migrations to Badgeyay

Badgeyay project is divided into two parts i.e front-end of Ember JS and back-end with REST-API programmed in Python.

We have integrated PostgreSQL as the object-relational database in Badgeyay and we are using SQLAlchemy SQL Toolkit and Object Relational Mapper tools for working with databases and Python. As we have Flask microframework for Python, so we are having Flask-SQLAlchemy as an extension for Flask that adds support for SQLAlchemy to work with the ORM.

One of the challenging jobs is to manage changes we make to the models and propagate these changes in the database. For this purpose, I have added Added Migrations to Flask SQLAlchemy for handling database changes using the Flask-Migrate extension.

In this blog, I will be discussing how I added Migrations to Flask SQLAlchemy for handling Database changes using the Flask-Migrate extension in my Pull Request.

First, Let’s understand Database Models, Migrations, and Flask Migrate extension. Then we will move onto adding migrations using Flask-Migrate. Let’s get started and understand it step by step.

What are Database Models?

A Database model defines the logical design and structure of a database which includes the relationships and constraints that determine how data can be stored and accessed. Presently, we are having a User and file Models in the project.

What are Migrations?

Database migration is a process, which usually includes assessment, database schema conversion. Migrations enable us to manipulate modifications we make to the models and propagate these adjustments in the database. For example, if later on, we make a change to a field in one of the models, all we will want to do is create and do a migration, and the database will replicate the change.

What is Flask Migrate?

Flask-Migrate is an extension that handles SQLAlchemy database migrations for Flask applications using Alembic. The database operations are made available through the Flask command-line interface or through the Flask-Script extension.

Now let’s add support for migration in Badgeyay.

Step 1 :

pip install flask-migrate

 

Step 2 :

We will need to edit run.py and it will look like this :

import os
from flask import Flask
from flask_migrate import Migrate  // Imported Flask Migrate

from api.db import db
from api.config import config

......

db.init_app(app)
migrate = Migrate(app, db) // It will allow us to run migrations
......

@app.before_first_request
def create_tables():
    db.create_all()

if __name__ == '__main__':
    app.run()

 

Step 3 :

Creation of Migration Directory.

 export FLASK_APP=run.py
 flask db init

 

This will create Migration Directory in the backend API folder.

└── migrations
    ├── README
    ├── alembic.ini
    ├── env.py
    ├── script.py.mako
    └── versions

 

Step 4 :

We will do our first Migration by the following command.

flask db migrate

 

Step 5 :

We will apply the migrations by the following command.

flask db upgrade

 

Now we are all done with setting up Migrations to Flask SQLAlchemy for handling database changes in the badgeyay repository. We can verify the Migration by checking the database tables in the Database.

This is how I have added Migrations to Flask SQLAlchemy for handling Database changes using the Flask-Migrate extension in my Pull Request.

Resources:

  • PostgreSQL Docs    – Link
  • Flask Migrate Docs  – Link
  • SQLAlchemy Docs  – Link
  • Flask SQLAlchemy Docs – Link
Continue ReadingImplementing Database Migrations to Badgeyay
Read more about the article PSLab Remote Lab: Automatically deploying the EmberJS WebApp and Flask API Server to different domains

PSLab Remote Lab: Automatically deploying the EmberJS WebApp and Flask API Server to different domains

The remote-lab software of the pocket science lab enables users to access their devices remotely via the internet. Its design involves an API server designed with Python Flask, and a web-app designed with EmberJS that allows users to access the API and carry out various tasks such as writing and executing Python scripts. For testing purposes, the repository needed to be setup to deploy both the backend as well as the webapp automatically when a build passes, and this blog post deals with how this can be achieved.

Deploying the API server

The Heroku PaaS was chosen due to its ease of use with a wide range of server software, and support for postgresql databases. It can be configured to automatically deploy branches from github repositories, and conditions such as passing of a linked CI can also be included. The following screenshot shows the Heroku configuration page of an app called pslab-test1. Most of the configuration actions can be carried out offline via the Heroku-Cli

 

In the above page, the pslab-test1 has been set to deploy automatically from the master branch of github.com/jithinbp/pslab-remote . The wait for CI to pass before deploy has been disabled since a CI has not been setup on the repository.

Files required for Heroku to deploy automatically

Once the Heroku PaaS has copied the latest commit made to the linked repository, it searches the base directory for a configuration file called runtime.txt which contains details about the language of the app and the version of the compiler/interpretor to use, and a Procfile which contains the command to launch the app once it is ready. Since the PSLab’s API server is written in Python, we also have a requirements.txt which is a list of dependencies to be installed before launching the application.

Procfile

web: gunicorn app:app –log-file –

runtime.txt

python-3.6.1

requirements.txt

gunicorn==19.6.0
flask >= 0.10.1
psycopg2==2.6.2
flask-sqlalchemy
SQLAlchemy>=0.8.0
numpy>=1.13
flask-cors>=3.0.0

But wait, our app cannot run yet, because it requires a postgresql database, and we did not do anything to set up one. The following steps will set up a postgres database using the heroku-cli usable from your command prompt.

  • Point Heroku-cli to our app
    $ heroku git:remote -a pslab-test1
  • Create a postgres database under the hobby-dev plan available for free users.
    $ heroku addons:create heroku-postgresql:hobby-dev

    Creating heroku-postgresql:hobby-dev on ⬢ pslab-test1… free
    Database has been created and is available
    ! This database is empty. If upgrading, you can transfer
    ! data from another database with pg:copy
    Created postgresql-slippery-81404 as HEROKU_POSTGRESQL_CHARCOAL_URL
    Use heroku addons:docs heroku-postgresql to view documentation

  • The previous step created a database along with an environment variable HEROKU_POSTGRESQL_CHARCOAL_URL . As a shorthand, we can also refer to it simply as CHARCOAL .
  • In order to make it our primary database, it must be promoted

    $ heroku pg:promote HEROKU_POSTGRESQL_CHARCOAL_URL
    The database will now be available via the environment variable DATABASE_URL

  • Further documentation on creating and modifying postgres databases on Heroku can be found in the articles section .

At this point, if the app is in good shape, Heroku will automatically deploy its contents to pslab-test1.herokuapp.com. We can test it using a developer tool such as Postman, or make our own webapp to use it.

Deploying the EmberJS WebApp

Since we are using the free plan on Heroku which only allows one dyno, our EmberJS webapp which shares the repository cannot be deployed on the same heroku server. Therefore, we must look for other domains where the frontend can be deployed.

Surge.sh allows easy deployment of Ember apps, and we shall set up our CI’s configuration file .travis.yml to do this for us when a pull request is made, and the build passes

This excerpt from .travis.yml only shows parts relevant to deployment on Surge.sh

after_success:
– pushd frontend
– bash surge_deploy.sh
– popd

Once the build has passed, the after_success hook executes a script called surge_deploy.sh which is located in the directory of the webapp.

Contents of surge_deploy.sh

#!/usr/bin/env bash
if [ “$TRAVIS_PULL_REQUEST” == “false” ]; then
echo “Not a PR. Skipping surge deployment”
exit 0
fi

ember build –environment=’production’

export REPO_SLUG_ARRAY=(${TRAVIS_REPO_SLUG//\// })
export REPO_OWNER=${REPO_SLUG_ARRAY[0]}
export REPO_NAME=${REPO_SLUG_ARRAY[1]}

npm i -g surge

# Details of a dummy account. So can be added to vcs.
export SURGE_LOGIN=j********r@gmail.com
export SURGE_TOKEN=4********************************f
export DEPLOY_DOMAIN=https://${REPO_NAME}.surge.sh
surge –project ./dist –domain $DEPLOY_DOMAIN;

The variables SURGE_LOGIN and SURGE_TOKEN must be specified, otherwise Surge will open a login prompt, and since there is no way to feed details into a prompt in a Travis build, it will timeout and fail. The surge token can be obtained with a simple `surge login` followed by `surge token` on your system’s terminal.

Final Application

A user’s homepage on the webapp deployed at pslab-remote.surge.sh . The EmberJS app has been configured to send all AJAX requests to the API server located at pslab-remote.herokuapp.com .

Resources
Continue ReadingPSLab Remote Lab: Automatically deploying the EmberJS WebApp and Flask API Server to different domains
Read more about the article Designing A Remote Laboratory With PSLab: execution of function strings

Designing A Remote Laboratory With PSLab: execution of function strings

In the previous blog post, we introduced the concept of a ‘remote laboratory’, which would enable users to access the various features of the PSLab via the internet. Many aspects of the project were worked upon, which also involved creation of a web-app using EmberJS that enables users to create accounts , sign in, and prepare Python programs to be sent to the server for execution. A backend APi server based on Python-flask was also developed to handle these tasks, and maintain a postgresql database using sqlalchemy .

The following screencast shows the basic look and feel of the proposed remote lab running in a web browser.

This blog post will deal with implementing a way for the remote user to submit a simple function string, such as get_voltage(‘CH1’), and retrieve the results from the server.

There are three parts to this:
  • Creating a dictionary of the functions available in the sciencelab instance. The user will only be allowed access to these functions remotely, and we may protect some functions as the initialization and destruction routines by blocking them from the remote user
  • Creating an API method to receive a form containing the function string, execute the corresponding function from the dictionary, and reply with JSON data
  • Testing the API using the postman chrome extension
Creating a dictionary of functions :

The function dictionary maps function names against references to the actual functions from an instance of PSL.sciencelab . A simple dictionary containing just the get_voltage function can be generated in the following way:

from PSL import sciencelab
I=sciencelab.connect()
functionList = {'get_voltage':I.get_voltage}

This dictionary is then used with the eval method in order to evaluate a function string:

result = eval('get_voltage('CH1')',functionList)
print (result)
0.0012

A more efficient way to create this list is to use the inspect module, and automatically extract all the available methods into a dictionary

functionList = {}
for a in dir(I):
	attr = getattr(I, a)
	if inspect.ismethod(attr) and a!='__init__':
		functionList[a] = attr

In the above, we have made a dictionary of all the methods except __init__

This approach can also be easily extrapolated to automatically generate a dictionary for inline documentation strings which can then be passed on to the web app.

Creating an API method to execute submitted function strings

We create an API method that accepts a form containing the function string and option that specifies if the returned value is to be formatted as a string or JSON data. A special case arises for numpy arrays which cannot be directly converted to JSON, and the toList function must first be used for them.

@app.route('/evalFunctionString',methods=['POST'])
def evalFunctionString():
    if session.get('user'):
        _stringify=False
        try:
            _user = session.get('user')[1]
            _fn = request.form['function']
            _stringify = request.form.get('stringify',False)
            res = eval(_fn,functionList)
        except Exception as e:
            res = str(e)
        #dump string if requested. Otherwise json array
        if _stringify:
            return json.dumps({'status':True,'result':str(res),'stringified':True})
        else:
            #Try to simply convert the results to json
            try:
                return json.dumps({'status':True,'result':res,'stringified':False})
            # If that didn't work, it's due to the result containing numpy arrays.
            except Exception as e:
                #try to convert the numpy arrays to json using the .toList() function
                try:
                    return json.dumps({'status':True,'result':np.array(res).tolist(),'stringified':False})
                #And if nothing works, return the string
                except Exception as e:
                    print( 'string return',str(e))
                    return json.dumps({'status':True,'result':str(res),'stringified':True})
    else:
        return json.dumps({'status':False,'result':'unauthorized access','message':'Unauthorized access'})
Testing the API using Postman

The postman chrome extension allows users to submit forms to API servers, and view the raw results. It supports various encodings, and is quite handy for testing purposes.Before executing these via the evalFunctionString method, user credentials must first be submitted to the validateLogin method for authentication purposes.

Here are screenshots of the test results from a ‘get_voltage(‘CH1’)’ and ‘capture1(‘CH1’,20,1)’ function executed remotely via postman.

 

Our next steps will be to implement the dialog box in the frontend that will allow users to quickly type in function strings, and fetch the resultant data

Resources:

 

Continue ReadingDesigning A Remote Laboratory With PSLab: execution of function strings
Read more about the article Designing a Remote Laboratory with PSLab using Python Flask Framework

Designing a Remote Laboratory with PSLab using Python Flask Framework

In the introductory post about remote laboratories, a general set of tools to create a framework and handle its various aspects was also introduced. In this blog post, we will explore the implementation of several aspects of the backend app designed with python-flask, and the frontend based on EmberJS. A clear separation of the frontend and backend facilitates minimal disruption of either sections due to the other.

Implementing API methods in Python-Flask

In the Flask web server, page requests are handled via ‘routes’ , which are essentially URLs linked to a python function. Routes are also capable of handling payloads such as POST data, and various return types are also supported.

We shall use an example to demonstrate how a Sign-Up request sent from the sign-up form in the remote lab frontend for PSLab is handled.

@app.route('/signUp',methods=['POST'])
def signUp():
	"""Sign Up for Virtual Lab

	POST: Submit sign-up parameters. The following must be present:
	 inputName : The name of your account. does not need to be unique
	 inputEmail : e-mail ID used for login . must be unique.
	 inputPassword: password .
	Returns HTTP 404 when data does not exist.
	"""
	# read the posted values from the UI
	_name = request.form['inputName']
	_email = request.form['inputEmail']
	_password = request.form['inputPassword']

	# validate the received values
	if _name and _email and _password:
		_hashed_password = generate_password_hash(_password)
		newUser = User(_email, _name,_hashed_password)
		try:
			db.session.add(newUser)
			db.session.commit()
			return json.dumps({'status':True,'message':'User %s created successfully. e-mail:%s !'%(_name,_email)})
		except Exception as exc:
			reason = str(exc)
			return json.dumps({'status':False,'message':str(reason)})

 

In this example, the first line indicates that all URL requests made to <domain:port>/signUp will be handled by the function signUp . During development, we host the server on localhost, and use the default PORT number 8000, so sign-up forms must be submitted to 127.0.0.1:8000/signUp .

For deployment on a globally accessible server, a machine with a static IP, and a DNS record must be used. An example for such a deployment would be the heroku subdomain where pslab-remote is automatically deployed ; https://pslab-stage.herokuapp.com/signUp

A closer look at the above example will tell you that POST data can be accessed via the request.form dictionary, and that the sign-up routine requires inputName,inputEmail, and inputPassword. A password hash is generated before writing the parameters to the database.

Testing API methods using the Postman chrome extension

The route described in the above example requires form data to be submitted along with the URL, and we will use a rather handy developer tool called Postman to help us do this. In the frontend apps , AJAX methods are usually employed to do such tasks as well as handle the response from the server.

 

The above screenshot shows Postman being used to submit form data to /signUp on our API server running at localhost:8000 . The fields inputName, inputDescription, and inputPassword are also posted along with it.

In the bottom section, one can see that the server returned a positive status variable, as well as a descriptive message.

Submitting the sign up form via an Ember controller.
  • Setting up a template
    We first need to set up a template that we shall call sign-up.hbs , and add the following form to it. This form contains the details essential for signing up , and its submit action is linked to an action called `signMeUp` . This action will be defined in the controller which we shall explore shortly

<form class="form-signin" {{action "signMeUp" on="submit"}} >
        <label for="inputName" class="sr-only">Your Name</label>
        {{input value=inputName type="text" name="inputName" id="inputName" class="form-control" placeholder="name" required=true autofocus=true}}
        <label for="inputEmail" class="sr-only">Email address</label>
        {{input value=inputEmail type="email" name="inputEmail" id="inputEmail" class="form-control" placeholder="Email address" required=true autofocus=true}}
        <label for="inputPassword" class="sr-only">Password</label>
        {{input value=inputPassword type="password" name="inputPassword" id="inputPassword" class="form-control" placeholder="Password" required=true autofocus=true}}
         
        <button class="btn btn-lg btn-primary btn-block" type="submit">Sign Up</button>
</form>

 

  • Defining the controller
    The controller contains the actions and variables that the template links to. In this case, we require an action called signMeUp. The success, failure, and error handlers are hidden for clarity.

import Ember from 'ember';
export default Ember.Controller.extend({
  actions:{
    signMeUp() {
        var request = Ember.$.post("/signUp",
 this.getProperties("inputName","inputEmail","inputPassword"),this,'json');
        request.then(this.success.bind(this), this.failure.bind(this),
this.error.bind(this));
    },
  },
});

The signMeUp action submits the contents of the form to the signUp route on the API server, and the results are handled by functions called success, failure, or error depending on the type of response from the backend server.

Resources:

 

Continue ReadingDesigning a Remote Laboratory with PSLab using Python Flask Framework