Rest API

Adding JSON-API to Badgeyay Backend

Badgeyay has two main components, the Python-Flask backend server and the EmberJS frontend.

EmberJS frontend uses ember data to save the data from the backend server api into the store of EmberJS frontend. To make the ember data frontend comply with backend api we need the backend server to send responses that comply with the standards of the JSON-API.

What is JSON-API?

As stated by JSONAPI.ORG

"If you've ever argued with your team about the way your JSON responses should be formatted, JSON API can be your anti-bikeshedding tool."

To put it up simply, JSON-API is a way of representing the JSON data that is being generated by the server backend. In this way we represent the JSON data in a particular way that follows the JSON-API convention. An example of data that follows json-api standards is given below:

{
"data": {
"id": "1",
"type": "posts",
"attributes": {
"title": "This is a JSON API data"
},
"relationships": {
"author": {
"links": {
"related": "/example/number"
}
},
"comments": {
"links": {
"related": "/example/number/article/"
}
"data": [
{"id": 5, "type": "example"},
{"id": 12, "type": "example"}
],
}
},
}
}

Adding JSON-API using Marshmallow-JSONAPI

We proceeded on to adding json-api into the Python-Flask backend. Before we proceed to adding json-api, we first need to install marshmallow_jsonapi

To install marshmallow_jsonapi

$ ~ pip install marshmallow-jsonapi

After installing marshmallow_jsonapi, we proceed onto making our first schema.

A schema is a layer of abstraction that is provided over a database model that can be used to dump data from or into an object. This object can therefore be used to either store in database or to dump it to the EmberJS frontend. Let us create a schema for File.

from marshmallow_jsonapi.flask import Schema
from marshmallow_jsonapi import fields


class FileSchema(Schema):
class Meta:
type_ = 'File'
self_view = 'fileUploader.get_file'
kwargs = {'id': '<id>'}

id = fields.Str(required=True, dump_only=True)
filename = fields.Str(required=True)
filetype = fields.Str(required=True)
user_id = fields.Relationship(
self_url='/api/upload/get_file',
self_url_kwargs={'file_id': '<id>'},
related_url='/user/register',
related_url_kwargs={'id': '<id>'},
include_resource_linkage=True,
type_='User'
)

So we have successfully created a Schema for getting files. This schema has an id, filename and filetype. It also has a relationship with the User.

Let us now create a route for this Schema. The below snippet of code is used to find a given file using this schema.

@router.route('/get_file', methods=['GET'])
def get_file():
input_data = request.args
file = File().query.filter_by(filename=input_data.get('filename')).first()
return jsonify(FileSchema().dump(file).data)

 

Now to get details of a file using our newly created route and schema all we need to do is use the following cURL command:

$ ~ curl -X GET "http://localhost:5000/api/upload/get_file?filename={your_file_name}"

You will get something like this as a response:

{
"data": {
"attributes": {
"filename": "13376967-8846-4c66-bcab-4a6b7d58aca7.csv",
"filetype": "csv"
},
"id": "967dc51b-289a-43a1-94c1-5cfce04b0fbf",
"links": {
"self": "/api/upload/get_file"
},
"relationships": {
"user_id": {
"data": {
"id": "J9v2LBIai1MOc8LijeLx7zWsP4I2",
"type": "User"
},
"links": {
"related": "/user/register",
"self": "/api/upload/get_file"
}
}
},
"type": "File"
},
"links": {
"self": "/api/upload/get_file"
}
}

Further Improvements

After adding JSON-API standards to the backend API we can easily integrate it with the EmberJS frontend. Now we can work on adding more schemas as a method of layers of abstraction so that the backend can serve more functionalities and the data can be consumed by the frontend as well.

Resources

Continue ReadingAdding JSON-API to Badgeyay Backend

Generating Badges from Badgeyay API

Badgeyay is a badge generator and its main functionality is generating badges. Since the beginning of GSoC 2018 period, Badgeyay is under refactoring and remodeling process. We have introduced many APIs to make sure that Badgeyay works. Now, the badge generator has an endpoint to generate badges for your events/meetups

How to create badges?

Creating badges using the newly formed API is simpler than before. All you need to do is pass some basic details of the image you want, the data you want, the size and the color of font etc to the API and woosh! Within a blink of your eye the badges are generated.

Backend requires some data fields to generate badges

{
"csv" : "a731h-jk12n-bbau2-saj2-nxg31.csv",
"image" : "p2ja7-gna398-c23ba-naj31a.png",
"text-color" : "#ffffff"
}

“csv” is the filename of the csv that the user uploads and get back as a result, “image” is the image name that user gets after a successful upload to the respective APIs, “text-color” is the color of the text that the user wants on the badges.

Output of the API

{
"output" :  "path-to-the-pdf-of-the-badge-generated",
.
.
}

What is happening behind the scene?

Once the user sends the data to the API, the required route is triggered and the data is checked,If the data is not present an error response is sent back to the user so as to inform them about the misplacement or improper format of data.

import os
from flask import Blueprint, jsonify, request
from flask import current_app as app
# from api.helpers.verifyToken import loginRequired
from api.utils.response import Response
from api.utils.svg_to_png import SVG2PNG
from api.utils.merge_badges import MergeBadges


router = Blueprint('generateBadges', __name__)


@router.route('/generate_badges', methods=['POST'])
def generateBadges():
try:
data = request.get_json()
except Exception as e:
return jsonify(
Response(401).exceptWithMessage(str(e),'Could not find any JSON'))

if not data.get('csv'):
return jsonify(
Response(401).generateMessage('No CSV filename found'))
if not data.get('image'):
return jsonify(Response(401).generateMessage('No Image filename found'))
csv_name = data.get('csv')
image_name = data.get('image')
text_color = data.get('text-color') or '#ffffff'
svg2png = SVG2PNG()
svg2png.do_text_fill('static/badges/8BadgesOnA3.svg', text_color)
merge_badges = MergeBadges(image_name, csv_name)
merge_badges.merge_pdfs()

output = os.path.join(app.config.get('BASE_DIR'), 'static', 'temporary', image_name)
return jsonify(
Response(200).generateMessage(str(output)))

 

After the data is received, we send it to MergeBadges which internally calls the GenerateBadges class which creates the badges.

Brief explanation of the Badge Generation Process:
- Gather data from the user- Fill the SVG for badges with the text color

- Load the image from uploads directory
- Generate badges for every individual
- Create PDFs for individual Badges
- Merge those PDFs to provide an all-badges pdf to the user

 

And this is how we generated badges for the user using the Badgeyay Backend API.

How is this effective?

We are making sure that the user chooses the image and csv that he/she has uploaded only,

In this way we maintain a proper workflow, we also manage these badges into the database and hence using the filenames helps a lot.It does not involve sending huge files and a lot of data like we had in the previous API.

Earlier, we used to send the image and the csv altogether that caused a serious mismanagement of the project. In this case we are accepting the CSVs and the Images on different API routes and then using the specific image and csv to make badges. We can now more easily relate to the files associated with each and every badge and henceforth we can easily manage them in the database.

Further Improvements

We will work on adding security to the route so that not anyone can create badges. We also need to integrate database into badges generated service so that we can maintain the badges that the user has generated.

Resources

Continue ReadingGenerating Badges from Badgeyay API

File and Image Upload API in Badgeyay

Badgeyay has seen many changes in the recent past during its refactoring. It started off with backend and we have now transition to remodeling backend as well.

The backend transition is working perfectly. We have established sufficient APIs so far to get it working.

Some of the most important APIs that we created are

  • Image Upload API
  • File Upload API

Why do we need APIs?

We need APIs so that the frontend written in Ember JS can coordinate with the backend written in Python Flask with the database being PostgreSQL.

Creating the APIs

Creating these APIs is easy and straightforward. The following APIs are written in Python Flask with a backend database support of PostgreSQL.

Image Upload API

The image upload API considers that the frontend is sending the Image as a base64 encoded string and the backend is supposed to accept this string and convert this string into an image and save it onto the server.

We proceed by creating a file named fileUploader.py and code the following API.

First of all, we need to declare the imports

from flask import Blueprint, request, jsonify
from api.utils.response import Response
from api.helpers.verifyToken import loginRequired
from api.helpers.uploads import saveToImage, saveToCSV

Now, let’s create a route for image upload.

router = Blueprint('fileUploader', __name__)

@router.route('/image', methods=['POST'])
@loginRequired
def uploadImage():
try:
image = request.json['data']
except Exception as e:
return jsonify(
Response(400).exceptWithMessage(
str(e),
'No Image is specified'))

extension = request.json['extension']
try:
imageName = saveToImage(imageFile=image, extension=extension)
except Exception as e:
return jsonify(
Response(400).exceptWithMessage(
str(e),
'Image could not be uploaded'))

return jsonify(
Response(200).generateMessage({
'message': 'Image Uploaded Successfully',
'unique_id': imageName}))

We are using the saveToImage function to actually save the image to the backend server.

The function definition of saveToImage function is given below.

def generateFileName():
return str(uuid.uuid4())def saveToImage(imageFile=None, extension='.png'):
imageName = generateFileName() + extension
imageDirectory = os.path.join(app.config.get('BASE_DIR'), 'static', 'uploads', 'image')if not os.path.isdir(imageDirectory):
os.makedirs(imageDirectory)imagePath = os.path.join(imageDirectory, imageName)
image = open(imagePath, "wb")
image.write(imageFile.decode('base64'))
image.close()

return imageName

Similarly, we are using file upload route to upload files to backend server.

The route for uploading files along with its helper function saveToCSV is given below.

def saveToCSV(csvFile=None, extension='.csv'):
csvName = generateFileName() + extension
csvDirectory = os.path.join(app.config.get('BASE_DIR'), 'static', 'uploads', 'csv')if not os.path.isdir(csvDirectory):
os.makedirs(csvDirectory)csvPath = os.path.join(csvDirectory, csvName)
csvFile.save(csvPath)return csvName
@router.route('/file', methods=['POST'])
@loginRequired
def fileUpload():
if 'file' not in request.files:
return jsonify(
Response(401).generateMessage(
'No file is specified'))file = request.files['file']
try:
csvName = saveToCSV(csvFile=file, extension='.csv')
except Exception as e:
return jsonify(
Response(400).exceptWithMessage(
str(e),
'CSV File could not be uploaded'))return jsonify(
Response(200).generateMessage({
'message': 'CSV Uploaded successfully',
'unique_id': csvName}))

What happens to the uploaded files?

The uploaded files gets saved into their respective directories, i.e. static/uploads/csv for CSV files and static/uploads/images for Image uploads.

The developer can view them from their respective folders. The static folder has been added to .gitignore  so that it does not gets uploaded to github repository.

Everything has been taken care of with immense accuracy and proper error handling.

Further Improvements

Further improvements in Badgeyay includes adding separate database models, work on adding a beautiful frontend and to add proper routes for completing the backend.

Resources

Continue ReadingFile and Image Upload API in Badgeyay

Unit Tests for REST-API in Python Web Application

Badgeyay backend is now shifted to REST-API and to test functions used in REST-API, we need some testing technology which will test each and every function used in the API. For our purposes, we chose the popular unit tests Python test suite.

In this blog, I’ll be discussing how I have written unit tests to test Badgeyay  REST-API.

First, let’s understand what is unittests and why we have chosen it. Then we will move onto writing API tests for Badgeyay. These tests have a generic structure and thus the code I mention would work in other REST API testing scenarios, often with little to no modifications.

Let’s get started and understand API testing step by step.

What is Unittests?

Unitests is a Python unit testing framework which supports test automation, sharing of setup and shutdown code for tests, aggregation of tests into collections, and independence of the tests from the reporting framework. The unittest module provides classes that make it easy to support these qualities for a set of tests.

Why Unittests?

We get two primary benefits from unit testing, with a majority of the value going to the first:

  • Guides your design to be loosely coupled and well fleshed out. If doing test driven development, it limits the code you write to only what is needed and helps you to evolve that code in small steps.
  • Provides fast automated regression for re-factors and small changes to the code.
  • Unit testing also gives you living documentation about how small pieces of the system work.

We should always strive to write comprehensive tests that cover the working code pretty well.

Now, here is glimpse of how  I wrote unit tests for testing code in the REST-API backend of Badgeyay. Using unittests python package and requests modules, we can test REST API in test automation.

Below is the code snippet for which I have written unit tests in one of my pull requests.

def output(response_type, message, download_link):
    if download_link == '':
        response = [
            {
                'type': response_type,
                'message': message
            }
        ]
    else:
        response = [
            {
                'type': response_type,
                'message': message,
                'download_link': download_link
            }
        ]
    return jsonify({'response': response})

 

To test this function, I basically created a mock object which could simulate the behavior of real objects in a controlled way, so in this case a mock object may simulate the behavior of the output function and return something like an JSON response without hitting the real REST API. Now the next challenge is to parse the JSON response and feed the specific value of the response JSON to the Python automation script. So Python reads the JSON as a dictionary object and it really simplifies the way JSON needs to be parsed and used.

And here’s the content of the backend/tests/test_basic.py file.

 #!/usr/bin/env python3
"""Tests for Basic Functions"""
import sys
import json
import unittest

sys.path.append("../..")
from app.main import *


class TestFunctions(unittest.TestCase):
      """Test case for the client methods."""
    def setup(self):
        app.app.config['TESTING'] = True
        self.app = app.app.test_client()
      # Test of Output function
    def test_output(self):
        with app.test_request_context():
            # mock object
            out = output('error', 'Test Error', 'local_host')
            # Passing the mock object
            response = [
                {
                    'type': 'error',
                    'message': 'Test Error',
                    'download_link': 'local_host'
                }
            ]
            data = json.loads(out.get_data(as_text=True))
            # Assert response
            self.assertEqual(data['response'], response)


if __name__ == '__main__':
    unittest.main()

 

And finally, we can verify that everything works by running nosetests .

This is how I wrote unit tests in BadgeYaY repository. You can find more of work here.

Resources:

  • The Purpose of Unit Testing – Link
  • Unit testing framework – Link
Continue ReadingUnit Tests for REST-API in Python Web Application

Parallelizing Builds In Travis CI

Badgeyay project is now divided into two parts i.e front-end of emberJS and back-end with REST-API programmed in Python. Now, one of the challenging job is that, it should support the uncoupled architecture. It should therefore run tests for the front-end and backend i.e, of two different languages on isolated instances by making use of the isolated parallel builds.

In this blog, I’ll be discussing how I have configured Travis CI to run the tests parallely in isolated parallel builds in Badgeyay in my Pull Request.

First let’s understand what is Parallel Travis CI build and why we need it. Then we will move onto configuring the travis.yml file to run tests parallely. Let’s get started and understand it step by step.

Why Parallel Travis CI Build?

The integration test suites tend to test more complex situations through the whole stack which incorporates front-end and back-end, they likewise have a tendency to be the slowest part, requiring various minutes to run, here and there even up to 30 minutes. To accelerate a test suite like that, we can split it up into a few sections utilizing Travis build matrix feature. Travis will decide the build matrix based on environment variables and schedule two builds to run.

Now our objective is clear that we have to configure travis.yml to build parallel-y. Our project requires two buildpacks, Python and node_js, running the build jobs for both them would speed up things by a considerable amount.It seems be possible now to run several languages in one .travis.yml file using the matrix:include feature.

Below is the code snippet of the travis.yml file  for the Badgeyay project in order to run build jobs in a parallel fashion.

sudo: required
dist: trusty

# check different combinations of build flags which is able to divide builds into “jobs”.
matrix:

# Helps to run different languages in one .travis.yml file
include:

# First Job in Python.
- language: python3

apt:
packages:
- python-dev

python:
- 3.5
cache:
directories:
- $HOME/backend/.pip-cache/

before_install:
- sudo apt-get -qq update
- sudo apt-get -y install python3-pip
- sudo apt-get install python-virtualenv

install:
- virtualenv  -p python3 ../flask_env
- source ../flask_env/bin/activate
- pip3 install -r backend/requirements/test.txt --cache-dir

before_script:
- export DISPLAY=:99.0
- sh -e /etc/init.d/xvfb start
- sleep 3

script:
- python backend/app/main.py >> log.txt 2>&1  &
- python backend/app/main.py > /dev/null &
- py.test --cov ../  ./backend/app/tests/test_api.py

after_success:
- bash <(curl -s https://codecov.io/bash)

# Second Job in node js.
- language: node_js
node_js:
- "6"

addons:
chrome: stable

cache:
directories:
- $HOME/frontend/.npm

env:
global:
# See https://git.io/vdao3 for details.
- JOBS=1

before_install:
- cd frontend
- npm install
- npm install -g ember-cli
- npm i eslint-plugin-ember@latest --save-dev
- npm config set spin false

script:
- npm run lint:js
- npm test

 

Now, as we have added travis.yml and pushed it to the project repo. Here is the screenshot of passing Travis CI after parallel build jobs.

The related PR of this work is https://github.com/fossasia/badgeyay/pull/512

Resources :

Travis CI documentation – Link

Continue ReadingParallelizing Builds In Travis CI

sTeam REST API Unit Testing

(ˢᵒᶜⁱᵉᵗʸserver) aims to be a platform for developing collaborative applications.
sTeam server project repository: sTeam.
sTeam-REST API repository: sTeam-REST

Unit Testing the sTeam REST API

The unit testing of the sTeam REST API is done using the karma and the jasmine test runner. The karma and the jasmine test runner are set up in the project repository.

The karma test runner : The main goal for Karma is to bring a productive testing environment to developers. The environment being one where they don’t have to set up loads of configurations, but rather a place where developers can just write the code and get instant feedback from their tests. Because getting quick feedback is what makes you productive and creative.

The jasmine test runner: Jasmine is a behavior-driven development framework for testing JavaScript code. It does not depend on any other JavaScript frameworks. It does not require a DOM. And it has a clean, obvious syntax so that you can easily write tests.

The karma and jasmine test runner were configured for the project and basic tests were ran. The angular js and angular mocks version in the local development repository was different. This had resulted into a new error been incorporated into the project repo. The ‘angular.element.cleanData is not a function’ error is thrown in the local development repository. This error happens when the local version of the angular.js and angular-mocks.js doesn’t match. The testing framework would test you if the versions f the two libraries is not the same.

The jasmine test runner can be accessed from the browser. The karma tests can be performed from the command line.

To access the jasmine test runner from the web browser, go to the url

http://localhost:7000/test/unit/runner.html

To run the karma test suite, run the following command

$ karma start

The unit tests of the sTeam REST service were done using jasmine. The unit tests were written in coffee script. The preprocessor to compile the files from coffee script to javascript is defined in the karma configuration file.

Jasmine Test RunnerJasmineRunner
Jasmine Test Failure

JasmineRunnerFailure

First a dummy pass case and a fail case is tested to check there are no errors in the test suite during the test execution.

The localstoragemodule.js which is used in the steam service is injected in the test module. Then the steam service version is tested.

describe 'Check version of sTeam-service', -> 
 		it 'should return current version', inject (version) -> 
 			expect(version).toEqual('0.1')

steam service should be injected in a global variable as the same service functions shall be tested while performing the remaining tests.
Then the steam service is injected and checked whether it exists or not.

beforeEach inject (_steam_) -> 
 		steam= _steam_ 
 	describe 'Check sTeam service injection', ->  
 		it 'steam service should exist', -> 
 			expect(steam).toBeDefined()

The sTeam service has both private and public functions. The private functions cannot be accessed from outside. The private functions defined in the sTeam service arehandle_request and headers.

describe 'Check sTeam service functions are defined.', ->  
 		describe ' Check the sTeam REST API private functions.', -> 
 			it 'steam service handle request function should exist', -> 
 				expect(steam.handle_request).toBeDefined() 
 			it 'steam service headers function should exist', -> 
 				expect(steam.headers).toBeDefined()

The public functions of the sTeam service are then tested.

describe 'Check sTeam service functions are defined.', ->  
 		describe ' Check the sTeam REST API public functions.', -> 
 			it 'steam service login function should exist', -> 
 				expect(steam.login).toBeDefined() 
 			it 'steam service loginp function should exist', -> 
 				expect(steam.loginp).toBeDefined() 
 			it 'steam service logout function should exist', -> 
 				expect(steam.logout).toBeDefined() 
 			it 'steam service user function should exist', -> 
 				expect(steam.user).toBeDefined() 
 			it 'steam service get function should exist', -> 
 				expect(steam.get).toBeDefined() 
 			it 'steam service put function should exist', -> 
 				expect(steam.put).toBeDefined() 
 			it 'steam service post function should exist', -> 
 				expect(steam.post).toBeDefined() 
 			it 'steam service delete function should exist', -> 
 				expect(steam.delete).toBeDefined()

The test suite written by Siddhant for the sTeam server was tested. The merging of the code from different branches which includes the work done during the course of GSoC 2016 will be merged subsequently.

Karma test runner

KarmaTestCase

Feel free to explore the repository. Suggestions for improvements are welcomed.

Checkout the FOSSASIA Idea’s page for more information on projects supported by FOSSASIA.

 

Continue ReadingsTeam REST API Unit Testing

sTeam API Endpoint Testing

(ˢᵒᶜⁱᵉᵗʸserver) aims to be a platform for developing collaborative applications.
sTeam server project repository: sTeam.
sTeam-REST API repository: sTeam-REST

sTeam API Endpoint Testing using Frisby

sTeam API endpoint testing is done using Frisby.  Frisby is a REST API testing framework built on node.js and Jasmine that makes testing API endpoints very easy, speedy and joyous.

Issue. Github Issue Github PR
sTeam-REST Frisby Test for login Issue-38 PR-40
sTeam-REST Frisby Tests Issue-41 PR-42

Write Tests

Frisby tests start with frisby.create with a description of the test followed by one of get, post, put, delete, or head, and ending with toss to generate the resulting jasmine spec test. Frisby has many built-in test helpers like expectStatus to easily test HTTP status codes, expectJSON to test expected JSON keys/values, and expectJSONTypes to test JSON value types, among many others.

// Registration Tests
frisby.create('Testing Registration API calls')
.post('http://steam.realss.com/scripts/rest.pike?request=register', {
email: "ajinkya007.in@gmail.com",
fullname: "Ajinkya Wavare",
group: "realss",
password: "ajinkya",
userid: "aj007"
}, {json: true})
.expectStatus(200)
.expectJSON({
"request-method": "POST",
"request": "register",
"me": restTest.testMe,
"__version": testRegistrationVersion,
"__date": testRegistrationDate
})
.toss();

The testMe, testRegistrationVersion and testRegistrationDate are the functions written in the rest_spec.js.

The frisby API endpoint tests have been written for testing the user login, accessing the user home directory, user workarea, user container, user document, user created image,  groups and subgroups.

The REST API url’s used for testing are described below. A payload consists of the user id and password.

Check if the user can login.

http://steam.realss.com/scripts/rest.pike?request=aj007

Test whether a user workarea exists or not. Here aj workarea has been created by the user.

http://steam.realss.com/scripts/rest.pike?request=aj007/aj

Test whether a user created container exists or not.

http://steam.realss.com/scripts/rest.pike?request=aj007/container

Test whether a user created document exists or not.

http://steam.realss.com/scripts/rest.pike?request=aj007/abc.pike

Test whether a user created image(object of any mime-type) inside a container exists or not.

http://steam.realss.com/scripts/rest.pike?request=aj007/container/Image.jpeg

Test whether a user created document exists or not. The group name and the subgroups can be queried.
eg. GroupName: groups, Subgroup: test.
The subgroup should be appended using “.” to the groupname.

http://steam.realss.com/scripts/rest.pike?request=groups.test

Here “groups” is a Groupname and “gsoc” is a subgroup of it.

http://ngtg.techgrind.asia/scripts/rest.pike?request=groups.gsoc

FrisbyTests

FrisbyTestCount

Unit Testing the sTeam REST API

The unit testing of the sTeam REST API is done using the karma and the jasmine test runner. The karma and the jasmine test runner are set up in the project repository.

The karma test runner : The main goal for Karma is to bring a productive testing environment to developers. The environment being one where they don’t have to set up loads of configurations, but rather a place where developers can just write the code and get instant feedback from their tests. Because getting quick feedback is what makes you productive and creative.

The jasmine test runner: Jasmine is a behavior-driven development framework for testing JavaScript code. It does not depend on any other JavaScript frameworks. It does not require a DOM. And it has a clean, obvious syntax so that you can easily write tests.

The karma and jasmine test runner were configured for the project and basic tests were ran. The angular js and angular mocks version in the local development repository was different. This had resulted into a new error been incorporated into the project repo.

Angular Unit-Testing: TypeError ‘angular.element.cleanData is not a function’

When angular and angular-mocks are not of the same version, these error occurs while running the tests. If the versions of the two javascript libraries don’t match your tests will be testing to you.

The jasmine test runner can be accessed from the browser. The karma tests can be performed from the command line. These shall be enhanced further during the course of work.

Feel free to explore the repository. Suggestions for improvements are welcomed.

Checkout the FOSSASIA Idea’s page for more information on projects supported by FOSSASIA.

Continue ReadingsTeam API Endpoint Testing

Generating Documentation and Modifying the sTeam-REST API

(ˢᵒᶜⁱᵉᵗʸserver) aims to be a platform for developing collaborative applications.
sTeam server project repository: sTeam.
sTeam-REST API repository: sTeam-REST

Documentation

Documentation is an important part of software engineering. Types of documentation include:

  1. Requirements – Statements that identify attributes, capabilities, characteristics, or qualities of a system. This is the foundation for what will be or has been implemented.
  2. Architecture/Design – Overview of software. Includes relations to an environment and construction principles to be used in design of software components.
  3. Technical – Documentation of code, algorithms, interfaces, and APIs.
  4. End user – Manuals for the end-user, system administrators and support staff.
  5. Marketing – How to market the product and analysis of the market demand.

Doxygen

Doxygen is the de facto standard tool for generating documentation from annotated C++ sources, but it also supports other popular programming languages such as C, Objective-C, C#, PHP, Java, Python, IDL (Corba, Microsoft, and UNO/OpenOffice flavors), Fortran, VHDL, Tcl, and to some extent D.
The Doxygen treats files of other languages as C/C++ and creates documentation for it accordingly.
sTeam documentation was tried to be created with the doxygen. But empty documentation was created due to the lack of the doxygen annotations used in the project.
 Doxygen doc generation.
DoxyGenTerminal
Doxygen Docs
DoxyDoc
The next way to create documentation was to make use of the autodoc utility provided by the Pike. The utility to generate docs was provided in the later versions of the Pike(>=8.0.155).
The autodoc files are generated and  later these are converted into  html pages. The commands used for generating the autodoc include:-
pike -x extract_autodoc /source
pike -x autodoc_to_html /src /opfile
The autodoc_to_html utility converts a single autodoc file to an html page. As a result a shell script was written to convert all the generated autodoc files to the html file.
docGenerator.sh
#!/bin/bash  

shopt -s globstar  
for filename in ./**/*.pike.xml; do  
    outputFile=doc/${filename#./}  
    outputFile=${outputFile%.xml}."html"  
    if [ -d $(dirname "./"$outputFile) ]; then  
      touch "./"$outputFile  
    else  
      mkdir -p $(dirname "./"$outputFile) && touch "./"$outputFile  
    fi  
    pike -x autodoc_to_html $filename "./"$outputFile  
done

Autodoc Documentation
AutoDocThe documentation generated by this was less informative and lacked the referrals to other classes and headers. The societyserver project was developed long back but the autodoc utility was introduced in the later versions of pike. As a result the source files lacked the autodoc tags which are required to generate a well informative documentation with bindings to other files.

Restructuring the sTeam-REST API

The sTeam-REST API project made use of the angular-seed to initiate the development during the early phases. However these files still existed in the project. This had lead to a pandemonium and created difficulty in understanding the project. The files had to be removed and the app was in dire need of a restructuring. The following issues have been reported and resolved.

Issue. Github Issue Github PR
sTeam-REST Issues Issues PR

The new UI can be seen below.

Home

Home

Register

Register

About

About

Testing the REST API

The functionality to run the tests using the npm test command was added to the project. Now the user can run the tests using these commands instead of the traditional approach of running the tests using jasmine-node and specifying the test directory. The domain name of the urls to which the request was sent was changed. The e2e tests and frisby tests were conducted successfully.

e2e Tests.

e2eTests

Frisby Tests

NPM tests

The next step would be to do add more tests for the REST API.

Feel free to explore the repository. Suggestions for improvements are welcomed.

Checkout the FOSSASIA Idea’s page for more information on projects supported by FOSSASIA.

Continue ReadingGenerating Documentation and Modifying the sTeam-REST API

sTeam REST API

(ˢᵒᶜⁱᵉᵗʸserver) aims to be a platform for developing collaborative applications.
sTeam server project repository: sTeam.
sTeam-REST API repository: sTeam-REST

REST Services

REST is the software architectural style of the World Wide Web. REST (Representational State Transfer) was introduced by Roy Fielding in his doctoral dissertation in 2000. Its purpose is to induce performance, scalability, simplicity, modifiability, visibility, portability, and reliability.It has client/server relationship with a uniform interface and is stateless. REST is most commonly associated with HTTP but it is not strictly related to it.

REST Principles

  • Resources : Each and every component is a resource.A resource is accessed by a common interface using HTTP standard methods.
  • Messages use HTTP methods like GET, POST, PUT, and DELETE.
  • Resource identification through URI: Resources are identified using URI. Resources are represented using JSON or XML.
  • Stateless interactions take place between the server and the client. No context is saved for the requests at the server.The client maintains the state of the session.

HTTP methods

The CRUD(create, retrieve, update and delete ) operations are performed using the HTTP methods.

GET

It is used to retrieve information. GET requests executed any number of times with the same parameters, the results would not change. This makes it idempotent. Partial or conditional requests can be sent. It is a read only type of operation.

Retrieve a list of users:

GET /api.example.com/UserService/users

POST

POST is usually used to create a new entity. It can also be used to update an existing entity. The request will have to do something with the entity provided in the URI.

Create a new user with an ID 2:

POST /api.example.com/UserService/users/2

PUT

PUT request is always idempotent. Executing the same request any number of times will not change the output. PUT can be used to create or update an existing entity.

Modify the user with an ID of 1:

PUT /api.example.com/UserService/users/1

PATCH

It is idempotent. PATCH requests only updates the specified fields of an entity.

Modify a user with an id of 1:

PATCH /api.example.com/UserService/users/1

DELETE

It can be asynchronous or a long-running request. It removes the resource. It can be removed immediately or at a later instance of time.

Delete a user with an ID of 1:

DELETE /api.example.com/UserService/users/1

 sTeam-REST API

Installing and activating the REST API

The REST API is developed as an application inside the sTeam server. This simplifies development quite a lot, as we don’t need to restart the server for every change. Instead just the API code gets updated and reloaded. It may eventually be integrated into the core, however the longterm plan is actually to move functionality out of the core, to make development easier.

To get the current version of the API clone the steam-rest repo into your home or to any place where you keep your development repos. Then change to the tools directory of your installation and run import-from-git.

git clone https://github.com/societyserver/steam-rest
cd steam-rest
git checkout origin/rest.pike
export steamrest=`pwd`
cd /usr/local/lib/steam/tools
./import-from-git.pike -u root $steamrest /

Note: The new import-from-git.pike script supports importing documents of all mime types.

It is important that the first import is done as root because the API code needs to run with root privileges and it will only do that if the object that holds the source is created as root.

Once the api code is loaded there are just a few tweaks needed to make it work.

We need to fix the mime-type, as the import script is not doing that yet.

OBJ("/sources/rest.pike")->set_attribute("DOC_MIME_TYPE", "source/pike");

Changing the mime type will change the class of the rest api script from Document to DocLpc.

> OBJ("/sources/rest.pike");                                               
(1) Result: 127.0.0.1:1900/rest.pike(#840,/classes/Document,17,source/pike)
> OBJ("/sources/rest.pike");                                               
(2) Result: 127.0.0.1:1900/rest.pike+(#840,/classes/DocLpc,529,source/pike,0 Instances, ({  }))

This takes a moment, check the type a few times until it’s done. Then instantiate an object from the source, give it a proper name, and move it to the /scripts/ container”

object rest = OBJ("/sources/rest.pike")->provide_instance();
rest->set_attribute("OBJ_NAME", "rest.pike");
rest->move(OBJ("/scripts/"));

Instantiating the object needs to be done as sTeam-root, in order for it to have permissions to run on behalf of other users.

Once this is done you are ready to start using the API.

sTeam-REST API tests

The project contains a set of examples and tests for the RESTful API for the sTeam server.

The code is written in coffee script and needs node.js only for coffeescript translation. Deployment can be done as static javascript files, and does not need any kind of dynamic server for the front-end. The back-end is a RESTful API written for the sTeam server as used by steam.realss.com

Development instructions

step 1: install node.js

http://nodejs.org/download/

step 2: clone the repository

git clone https://github.com/societyserver/steam-rest

step 3: install node packages:

npm install

This installs all dependencies (including coffee) for our project into the project’s node_modules directory based on the ‘package.json’ file

step 4: start the server

node_modules/.bin/coffee scripts/server.coffee

but for convenience we can install coffee in the global node environment:

npm install -g coffee-script

so we can just say

coffee scripts/server.coffee

if the server is working you’ll see:

Listening on port 7000

Testing

FrisbyJS is used to test the API. It is run through Jasmine and is based on nodejs.

Once you have nodejs installed, run the following statement to install Frisby and Jasmine:

npm install -g jasmine-node frisby

Then execute the test by:

cd project/directory
jasmine-node test/

The karma testing framework is also used for testing the sTeam REST API.

There were some inherent issues with the test framework which were addressed.

Issue. Github Issue Github PR
Update Readme.md Update Readme PR-2
Add javascript dependencies Issue-4 PR-6
Add node dependencies Issue-5 PR-7
Add angular-mocks.js script for testing the REST services. Issue-8 PR-9

The project dependencies were not met and this resulted into the error when the project was run on the localhost. The angular-ui-router, angular-bootstrap and bootstrap js frameworks were not installed in the node modules of the project. As a result the bower.json script was modified to include these dependencies.

bower.json

{
  "name": "bower",
  "version": "0.1",
  "private": true,
  "ignore": [
    "**/.*",
    "node_modules",
    "bower_components",
    "test",
    "tests"
  ],
  "dependencies": {
    "angular": "",
    "angular-route": "~1.4.8",
    "angular-ui-router": "",
    "angular-bootstrap": "",
    "bootstrap": ""
  }
}

The node dependencies of karma, frisby and jasmine-node were included in the package.json. These would be installed when the npm install is executed.

package.json

{
"name": "TechGrind",
"version": "0.1.1",
"private": true,
"dependencies": {
"express": "",
"coffee-script": "",
"morgan": "",
"compression": "",
"method-override": "",
"body-parser": "",
"serve-static": "",
"errorhandler": "",
"bower": "",
"jasmine-node": "",
"frisby": "",
"karma": ""
},
"production_dirs": {
"coffee_src": "src/",
"src": "app/",
"dest": "app_production/"
},
"devDependencies": {
},
"scripts": {
"postinstall": "bower install"
}
}

Feel free to explore the repository. Suggestions for improvements are welcomed.

Checkout the FOSSASIA Idea’s page for more information on projects supported by FOSSASIA.

Continue ReadingsTeam REST API

Promises, REST API’s & Angular JS

week2gsoc1

Before breaking it down, to all those who don’t know what REST API’s are :

“REST is acronym for REpresentational State Transfer. It is architectural style for distributed hypermedia systems ans was first presented by Roy Fielding in 2000 in his famous dissertation.”

Source : “restfulapi.net”

So what is a Promise ?

Generally promises are representation of a value that which might not be available to us as of now but at some point eventually it will be available. Observe the below image to understand how promises work.

week2gsoc2

So how is this better than callbacks ?

Generally if you are using callbacks in your codebase then at some point of time either for i/o’s or file buffers or something might come in your way and makes you write nested callbacks. Writing nested callbacks is difficult and an alternative to that is promise.

week2gsoc3

REST API’s, promises what are you about to explain ?

Angular has this beautiful service called $q which is helpful for creating promises and using them in your angular applications. Generally many people have concerns over $q but $q is tightly integrated with the scope life cycle. So be it any task it contains all the necessary features needed by most of the heavy duty asynchronous tasks.

Now to start off angular can be really powerfull with the $http Service provided the fact that it has immense potential in allowing us to create heavy duty REST ful API’s, CRUD operation based web applications etc. So let us understand the $http service first before using promises for creating a REST ful API.

$http
|_____ .success()
For a success(), callback is called asynchronously when the request completes and the response arrives from the server
|_____ .error()
For an error(), the error callback is fired

Callbacks which are accpeted by above methods are :

  • data: It is a response obtained from server.
  • status: HTTP status codes 100x, 200x, 300x, 400x, etc which returned from server.
  • config: Object used for generating the request.
  • headers: Headers sent by the server.
  • statusText: HTTP status text of the status code.

Promise Chaining

Chaining is one of the most important aspects of promises. Here the basic idea is that promises can be composed through chaining. This means you can trigger a specified task only after previous promise has been resolved.

Handling errors by designing a promise that always rejects

No API is complete without writing a handler or a middleware that takes care of all the requests which are generally errored. Designing this will be useful if we want to tell a user about the occurrence of an error in terms of a promise rather than a simple value.

week2gsoc4

NOTE : Generally while using $http Service there are somethings which one should have knowledge of, If we happen to send a JS object as POST data in a POST/PUT request, it’s serialized to JSON before getting sent. Similarly, if the server responds with a JSON string it’s parsed into a JavaScript object and passed to the success callback attached to the promise.

Thats it folks,
Happy Hacking !!

Continue ReadingPromises, REST API’s & Angular JS