Open Event Web App – A PWA

Introduction

Progressive Web App (PWA) are web applications that are regular web pages or websites but can appear to the user like traditional applications or native mobile applications. The application type attempts to combine features offered by most modern browsers with the benefits of mobile experience. Open Event web app is a web application generator which has now introduced this new feature in its generated applications.

 

Why Progressive Web Apps?

The reasons why we enabled this functionality are that PWAs are –

  • Reliable – Load instantly and never show the downasaur, even in uncertain network conditions.
  • Fast – Respond quickly to user interactions with silky smooth animations and no janky scrolling.
  • Engaging – Feel like a natural app on the device, with an immersive user experience.

Thus where Open Event Web app generated applications are informative and only requires one time loading with functionalities like bookmarks depending on local storage of browser, we found Progressive web apps perfect to explain and demonstrate these applications as a whole.

How PWAs work?

The components associated with a progressive web application are :

Manifest: The web app manifest is a W3C specification defining a JSON-based manifest to provide developers a centralized place to put metadata associated with a web application.

Service Workers: Service Workers provide a scriptable network proxy in the web browser to manage the web/HTTP requests programmatically. The Service Workers lie between the network and device to supply the content. They are capable of using the cache mechanisms efficiently and allow error-free behavior during offline periods.

How we turned Open event Web app to a PWA?

Adding manifest.json

 {
"icons": [
    {
      "src": "./images/logo.png",
      "type": "image/png",
      "sizes": "96x96"
    }
  ],
  "start_url": "index.html",
  "scope": ".",
  "display": "standalone",
  "orientation": "portrait-primary",
  "background_color": "#fff",
  "theme_color": "#3f51b5",
  "description": "Open Event Web Application Generator",
  "dir": "ltr",
  "lang": "en-US"
}

 

Adding service workers

The initialization of service workers is done by calling an event listener namely ‘install’ :

var urlsToCache = [
 './css/bootstrap.min.css',
 './offline.html',
 './images/avatar.png'
];

self.addEventListener('install', function(event) {
 event.waitUntil(
   caches.open(CACHE_NAME).then(function(cache) {
     return cache.addAll(urlsToCache);
   })
 );
});

The service workers fetch the data from the cache when event listener ‘fetch’ is triggered. When a cache hit occurs the response data  is sent to the client from there otherwise it tries to fetch the data by making a request to the network. In case when network does not send response status code ‘200’, it sends an error response otherwise caches the data received.

self.addEventListener('fetch', function(event) {
 event.respondWith(
   caches.match(event.request).then(function(response) {
     // Cache hit - return response
     if (response) {
       return response;
     }

     var fetchRequest = event.request.clone();

     return fetch(fetchRequest)
       .then(function(response) {
         if (
           !response ||
           response.status !== 200 ||
           response.type !== 'basic'
         ) {
           return response;
         }
         var responseToCache = response.clone();
         caches.open(CACHE_NAME).then(function(cache) {
           cache.put(event.request, responseToCache);
         });
         return response;
       })
       .catch(function(err) {
         if (event.request.headers.get('Accept').indexOf('text/html') !== -1) {
           return caches.match('./offline.html');
         } else if (event.request.headers.get('Accept').indexOf('image') !== -1) {
           return caches.match('./images/avatar.png');
         } else {
           console.log(err);
         }
       });
   })
 );
});

The service workers are activated through the event listener namely ‘activate’ :

self.addEventListener('activate', function(event) {
 event.waitUntil(
   caches.keys().then(function(cacheNames) {
     return Promise.all(
       cacheNames.map(function(cacheName) {
         if (cacheName !== CACHE_NAME) {
           console.log('Deleting cache ' + cacheName);
           return caches.delete(cacheName);
         }
       })
     );
   })
 );
});

Adding service workers and manifest to the generator

Since we need to add the service workers and manifest to every web application generated through app generator, we copy the files ‘sw.js’ and ‘manifest.json’ in the directory structure of that particular web app using filestream module with the help of two abstract functions ‘copyServiceWorker’ and ‘copyManifestFile’ present in ‘distHelper.js’ code file.

 

distHelper.copyServiceWorker(appFolder, hashObj['hash'], function (err) {
 if (err) {
   console.log(err);
   logger.addLog('Error', 'Error occurred while copying service worker file', socket, err);
   return done(err);
 }
 return done(null);
});

distHelper.copyManifestFile(appFolder, eventName, function(err) {
 if (err) {
   console.log(err);
   logger.addLog('Error', 'Error occured while copying manifest file', socket, err);
   return done(err);
 }
 return done(null);
});

 

Further Improvements

Enabling push notifications for the bookmarked tracks and sessions. The user would be notified about the upcoming events through the notifications in the way the native mobile applications do.

Resources

Tags

GsoC’18, Fossasia, Eventyay, Open Event Web App, Progressive Web App

 

Unit Tests for REST-API in Python Web Application

Badgeyay backend is now shifted to REST-API and to test functions used in REST-API, we need some testing technology which will test each and every function used in the API. For our purposes, we chose the popular unit tests Python test suite.

In this blog, I’ll be discussing how I have written unit tests to test Badgeyay  REST-API.

First, let’s understand what is unittests and why we have chosen it. Then we will move onto writing API tests for Badgeyay. These tests have a generic structure and thus the code I mention would work in other REST API testing scenarios, often with little to no modifications.

Let’s get started and understand API testing step by step.

What is Unittests?

Unitests is a Python unit testing framework which supports test automation, sharing of setup and shutdown code for tests, aggregation of tests into collections, and independence of the tests from the reporting framework. The unittest module provides classes that make it easy to support these qualities for a set of tests.

Why Unittests?

We get two primary benefits from unit testing, with a majority of the value going to the first:

  • Guides your design to be loosely coupled and well fleshed out. If doing test driven development, it limits the code you write to only what is needed and helps you to evolve that code in small steps.
  • Provides fast automated regression for re-factors and small changes to the code.
  • Unit testing also gives you living documentation about how small pieces of the system work.

We should always strive to write comprehensive tests that cover the working code pretty well.

Now, here is glimpse of how  I wrote unit tests for testing code in the REST-API backend of Badgeyay. Using unittests python package and requests modules, we can test REST API in test automation.

Below is the code snippet for which I have written unit tests in one of my pull requests.

def output(response_type, message, download_link):
    if download_link == '':
        response = [
            {
                'type': response_type,
                'message': message
            }
        ]
    else:
        response = [
            {
                'type': response_type,
                'message': message,
                'download_link': download_link
            }
        ]
    return jsonify({'response': response})

 

To test this function, I basically created a mock object which could simulate the behavior of real objects in a controlled way, so in this case a mock object may simulate the behavior of the output function and return something like an JSON response without hitting the real REST API. Now the next challenge is to parse the JSON response and feed the specific value of the response JSON to the Python automation script. So Python reads the JSON as a dictionary object and it really simplifies the way JSON needs to be parsed and used.

And here’s the content of the backend/tests/test_basic.py file.

 #!/usr/bin/env python3
"""Tests for Basic Functions"""
import sys
import json
import unittest

sys.path.append("../..")
from app.main import *


class TestFunctions(unittest.TestCase):
      """Test case for the client methods."""
    def setup(self):
        app.app.config['TESTING'] = True
        self.app = app.app.test_client()
      # Test of Output function
    def test_output(self):
        with app.test_request_context():
            # mock object
            out = output('error', 'Test Error', 'local_host')
            # Passing the mock object
            response = [
                {
                    'type': 'error',
                    'message': 'Test Error',
                    'download_link': 'local_host'
                }
            ]
            data = json.loads(out.get_data(as_text=True))
            # Assert response
            self.assertEqual(data['response'], response)


if __name__ == '__main__':
    unittest.main()

 

And finally, we can verify that everything works by running nosetests .

This is how I wrote unit tests in BadgeYaY repository. You can find more of work here.

Resources:

  • The Purpose of Unit Testing – Link
  • Unit testing framework – Link

Parallelizing Builds In Travis CI

Badgeyay project is now divided into two parts i.e front-end of emberJS and back-end with REST-API programmed in Python. Now, one of the challenging job is that, it should support the uncoupled architecture. It should therefore run tests for the front-end and backend i.e, of two different languages on isolated instances by making use of the isolated parallel builds.

In this blog, I’ll be discussing how I have configured Travis CI to run the tests parallely in isolated parallel builds in Badgeyay in my Pull Request.

First let’s understand what is Parallel Travis CI build and why we need it. Then we will move onto configuring the travis.yml file to run tests parallely. Let’s get started and understand it step by step.

Why Parallel Travis CI Build?

The integration test suites tend to test more complex situations through the whole stack which incorporates front-end and back-end, they likewise have a tendency to be the slowest part, requiring various minutes to run, here and there even up to 30 minutes. To accelerate a test suite like that, we can split it up into a few sections utilizing Travis build matrix feature. Travis will decide the build matrix based on environment variables and schedule two builds to run.

Now our objective is clear that we have to configure travis.yml to build parallel-y. Our project requires two buildpacks, Python and node_js, running the build jobs for both them would speed up things by a considerable amount.It seems be possible now to run several languages in one .travis.yml file using the matrix:include feature.

Below is the code snippet of the travis.yml file  for the Badgeyay project in order to run build jobs in a parallel fashion.

sudo: required
dist: trusty

# check different combinations of build flags which is able to divide builds into “jobs”.
matrix:

# Helps to run different languages in one .travis.yml file
include:

# First Job in Python.
- language: python3

apt:
packages:
- python-dev

python:
- 3.5
cache:
directories:
- $HOME/backend/.pip-cache/

before_install:
- sudo apt-get -qq update
- sudo apt-get -y install python3-pip
- sudo apt-get install python-virtualenv

install:
- virtualenv  -p python3 ../flask_env
- source ../flask_env/bin/activate
- pip3 install -r backend/requirements/test.txt --cache-dir

before_script:
- export DISPLAY=:99.0
- sh -e /etc/init.d/xvfb start
- sleep 3

script:
- python backend/app/main.py >> log.txt 2>&1  &
- python backend/app/main.py > /dev/null &
- py.test --cov ../  ./backend/app/tests/test_api.py

after_success:
- bash <(curl -s https://codecov.io/bash)

# Second Job in node js.
- language: node_js
node_js:
- "6"

addons:
chrome: stable

cache:
directories:
- $HOME/frontend/.npm

env:
global:
# See https://git.io/vdao3 for details.
- JOBS=1

before_install:
- cd frontend
- npm install
- npm install -g ember-cli
- npm i [email protected] --save-dev
- npm config set spin false

script:
- npm run lint:js
- npm test

 

Now, as we have added travis.yml and pushed it to the project repo. Here is the screenshot of passing Travis CI after parallel build jobs.

The related PR of this work is https://github.com/fossasia/badgeyay/pull/512

Resources :

Travis CI documentation – Link

Deploying BadgeYaY with Docker on Docker Cloud

We already have a Dockerfile present in the repository but  there is problem in many lines of code.I studied about Docker and learned how It is deployed and I am now going to explain how I deployed BadgeYaY on Docker Cloud.

To make deploying of Badgeyay easier we are now supporting Docker based installation.

Before we start to deploy, let’s have a quick brief about what is docker and how it works ?

What is Docker ?

Docker is an open-source technology that allows you create, deploy, and run applications using containers. Docker allows you deploy technologies with many underlying components that must be installed and configured in a single, containerized instance.Docker makes it easier to create and deploy applications in an isolated environment.

Now, let’s start with how to deploy on docker cloud:

Step 1 – Installing Docker

Get the latest version of docker. See the offical site for installation info for your platform.

Step 2 – Create Dockerfile

With Docker, we can just grab a portable Python runtime as an image, no installation necessary. Then, our build can include the base Python image right alongside our app code, ensuring that our app, its dependencies, and the runtime, all travel together.

These portable images are defined by something called a Dockerfile.

In DockerFile, there are all the commands a user could call on the command line to assemble an image. Here’s is the Dockerfile of BadgeYaY.

# The FROM instruction initializes a new build stage and sets the Base Image for subsequent instructions.
FROM python:3.6

# We copy just the requirements.txt first to leverage Docker cache
COPY ./app/requirements.txt /app/


# The WORKDIR instruction sets the working directory for any RUN, CMD, ENTRYPOINT, COPY and ADD instructions that follow it in the Dockerfile.
WORKDIR /app


# The RUN instruction will execute any commands in a new layer on top of the current image and commit the results.
RUN pip install -r requirements.txt


# The COPY instruction copies new files.
COPY . /app


# An ENTRYPOINT allows you to configure a container that will run as an executable.
ENTRYPOINT [ "python" ]

# The main purpose of a CMD is to provide defaults for an executing container.
CMD [ "main.py" ]

 

Step 3 – Build New Docker Image

sudo docker build -t badgeyay:latest .

 

When the command completed successfully, we can check the new image with the docker command below:

     sudo docker images

 

Step 4 – Run the app

Let’s run the app in the background, in detached mode:

 sudo docker run -d -p 5000:5000 badgeyay

 

We get the long container ID for our app and then are kicked back to our terminal.Our container is running in the background.Now use docker container stop to end the process, using the CONTAINER ID, like so :

 

docker container stop 1fa4ab2cf395

 

Step 5 – Publish the app.

Log in to the Docker public registry on your local machine.

docker login

 

Upload your tagged image to the repository:

docker push username/repository:tag

 

From now on, we can use docker run and run our app on any machine. No matter where docker run executes, it pulls your image, along with Python and all the dependencies from requirements.txt, and runs your code. It all travels together in a neat little package, and the host machine doesn’t have to install anything but Docker to run it.

Docker Cloud

Docker Cloud provides a hosted registry service with build and testing facilities for Dockerized application images; tools to help you set up and manage host infrastructure; and application lifecycle features to automate deploying (and redeploying) services created from images.

In BadgeYaY, we  also have a Deploy button button which directly deploys on Docker cloud with a single click .

The related PR of this work is https://github.com/fossasia/badgeyay/pull/401 .

Resources :

  • Docker documentation: Link
  • Get Started With Docker: Link

Setting up Codecov in Badgeyay

 

BadgeYaY already has Travis CI and Codacy to test code quality and Pull Request but there was no support for testing Code Coverage in repository against every Pull Request. So I decided to go with setting up Codecov to test the code coverage.

In this blog post, I’ll be discussing how I have set up codecov in BadgeYaY in my Pull Request.

First, let’s understand what is codecov and why do we need it. For that we have to first understand what is code coverage then we will move on to how to add Codecov with help of Travis CI .

Let’s get started and understand it step by step.

What is Code Coverage ?

Code coverage is a measurement used to express which lines of code were executed by a test suite. We use three primary terms to describe each lines executed.

  • hit indicates that the source code was executed by the test suite.
  • partial indicates that the source code was not fully executed by the test suite; there are remaining branches that were not executed.
  • miss indicates that the source code was not executed by the test suite.

Coverage is the ratio of hits / (hit + partial + miss). A code base that has 5 lines executed by tests out of 12 total lines will receive a coverage ratio of 41% . In BadgeYaY , Code Coverage is 100%.

How CodeCov helps in Code Coverage ?

Codecov focuses on integration and promoting healthy pull requests. Codecov delivers <<<or “injects”>>> coverage metrics directly into the modern workflow to promote more code coverage, especially in pull requests where new features and bug fixes commonly occur.

I am listing down top 5 Codecov Features:

We can change the configuration of how Codecov processes reports and expresses coverage information. Let’s see how we configure it according to BadgeYaY by integrating it with Travis CI.

Now generally, the codecov works better with Travis CI. With the one line

 bash <(curl -s https://codecov.io/bash)

 

the code coverage can now be easily reported.

Add a script for testing:

"scripts": {
   - nosetests app/tests/test.py -v --with-coverage
}

Here is a particular example of travis.yml from the project repository of BadgeYaY:

Script:
- python app/main.py >> log.txt 2>&1  &
- nosetts app/tests/test.py -v --with-coverage
- python3 -m pyflakes

after_success:
- bash <(curl -s https://codecov.io/bash)

 

Let’s have a look at Codecov.yml to check exact configuration that I have used for BadgeYaY.

Codecov:
  # yes: will delay sending notifications until all ci is finished
  notify:
    require_ci_to_pass: yes

coverage:
  # how many decimal places to display in the UI: 0 <= value <= 4
  precision: 2
  # how coverage is rounded: down/up/nearest
  round: down 
  # custom range of coverage colors from red -> yellow -> green 
  range: "70...100"

  status:
     # measuring the overall project coverage
    project: yes
     # pull requests only: this commit status will measure the
       entire pull requests Coverage Diff. Checking if the lines
       adjusted are covered at least X%.
    patch: yes
     # if there are any unexpected changes in coverage
    changes: no

Comment:

  layout: "reach, diff, flags, files, footer"
  behavior: default
  require_changes: no

 

Now when anyone makes a Pull Request to BadgeYaY, Codecov will analyze the Pull Request according to above configuration and generate a Report showing the code coverage of that Pull Request.

 

Below is the screenshot of all test passing in BadgeYaY repository

This is how we setup codecov in BadgeYaY repository. And like this way, it can be set up in other repositories as well.

The related PR of this work is https://github.com/fossasia/badgeyay/pull/400

Resources :

  • CodeCov Documentation – Link