Implementing job queue in Open Event Web app

Open Event Web app enables multiple request handling by the implementation of queue system in its generator. Every request received from the client is saved and stored in the queue backed by redis server. The jobs are then processed one at a time using the FCFS (First come First Serve) job scheduling algorithm. Processing the requests one by one prevents the crashing of app and also prevents the loss of requests from the client.

Initialising job queue

The job queue is initialised with a name and the connection object of redis server as the arguments.

const redisClient =  require('redis').createClient(process.env.REDIS_URL);
const Queue = require('bee-queue');
const queue = new Queue('generator-queue', {redis: redisClient});

Handling jobs in queue

The client emits an event namely ‘live’ when request for event generation is received, the corresponding event is listened and a new job for the request is created and enqueued in the job queue. Every request received by the client is saved to ensure that there is no loss of request. The queue is then searched for the requests or the jobs which are in ‘waiting’ state, if the current request status for the job Id is waiting the socket emits an event namely ‘waiting’.

socket.on('live', function(formData) {
 const req = {body: formData};
 const job = queue.createJob(req);

 job.on('succeeded', function() {
   console.log('completed job ' + job.id);
 });

 job.save(async function(err, currentJob) {
   if (err) {
     console.log('job failed to save');
   }
   emitter = socket;
   console.log('saved job ' + currentJob.id);
   const jobs = await queue.getJobs('waiting', {start: 0, end: 25});
   const jobIds = await jobs.map((currJob) => currJob.id);

   if(jobIds.indexOf(currentJob.id) !== -1) {
     socket.emit('waiting');
   }
 });

});

Updating the status of request

If the socket emits the event ‘waiting’ it signifies that some other job is currently in process and the status of the current request is ‘waiting’.

socket.on('waiting', function () {
 updateStatusAnimate('Request status: Waiting');
});

Processing the jobs

When the queue is in ready state and no job is currently in process, it starts processing the saved job. The job is not completed until it receives a callback. The generator starts generating the event when the processing of request starts.

queue.on('ready', function() {
 queue.process(function(job, done) {
   console.log('processing job ' + job.id);
   generator.createDistDir(job.data, emitter, done);
 });
 console.log('processing jobs...');
});

 

The generator calls the callback function for the current job when the event generation completes or it is halted in between due to some error. As soon as the current job completes, next job in the queue starts being processed.

generator.createDistDir() = function(req, socket, callback){
  
  .....
  .....
  .....

  mailer.uploadAndsendMail(req.body.email, eventName, socket, (obj) => {
    if(obj.mail)
      logger.addLog('Success', 'Mail sent succesfully', socket);
    else
      logger.addLog('Error', 'Error sending mail', socket);

    if(emit) {
      socket.emit('live.ready', {
        appDir: appFolder,
        url: obj.url
      });
      callback(null);
    }
    else {
      callback(appFolder);
    }

    done(null, 'write');
   });
}

Resources

Continue ReadingImplementing job queue in Open Event Web app

Parallelizing travis build in Open Event Web app

 

Open Event Web app uses Travis CI as a platform to perform unit testing. Travis CI is a hosted, distributed continuous integration service used to build and test projects hosted at GitHub. Travis CI automatically detects when a commit has been made and pushed to a GitHub repository that is using Travis CI, and each time this happens, it will try to build the project and run tests. Travis build took around 24 minutes to complete when any commit is made to the project, which is a very large time, to reduce the build time we parallelized the build which uses maximum amount of resources available at that time and run the builds parallely which resulted into better use of resources as well as required lesser amount of time.

Open Event web app uses saucelabs integration to perform selenium tests and travis to perform continuous integration.

Why parallelize the build?

When there are unit tests that are independent of each other and can be executed using a common set of dependencies, those procedures can be performed parallely on different virtual machines bringing out maximum throughput.

Running say a large number of tests on a single machine can increase the build time to a large extent, this build time can be reduced significantly by running the tests parallely on different machines. Open Event Webapp has a build time of around 24 minutes which is reduced to half on parallelising the build.

Parallelizing your builds across virtual machines

To speed up a test suite, you can break it up into several parts using Travis CI’s build matrix feature.

Say you want to split up your unit tests and your integration tests into two different build jobs. They’ll run in parallel and fully utilize the available build capacity and the resources.

The architecture of open event webapp supports test suite for all the pages in the generated application. To parallelize the build, the test suite is divided in different files  with the directory structure as shown below:

   
   ├── test
      ├── serverTest.js
      ├── roomsAndSpeakers.js
      ├── tracks.js
      ├── generatorAndSchedule.js
      ├── sessionAndEvent,js

 

The env key in travis.yml is modified as shown below:

env:
  - TESTFOLDER=test/serverTest.js
  - TESTFOLDER=test/roomsAndSpeakers.js
  - TESTFOLDER=test/tracks.js
  - TESTFOLDER=test/generatorAndSchedule.js
  - TESTFOLDER=test/sessionAndEvent.js

 

The script running tests fetches environment variable and runs the test file accordingly as shown below:

# installing required items for build
install:
 - npm install -g istanbul mocha@3
 - npm install
 - npm install --save-dev

# testing script
script:
 - istanbul cover _mocha -- $TESTFOLDER

# notify codecov and deploy to cloud
after_success:
 if ([ "$TESTFOLDER" == "test/serverTest.js" ]); then
   bash <(curl -s https://codecov.io/bash);
   bash gh_deploy.sh && kubernetes/travis/deploy.sh;
 fi

Results:

The build time which was earlier 23 minutes is reduced to 12 minutes after parallelizing the build.

Resources

 

Continue ReadingParallelizing travis build in Open Event Web app

Enable web app generation for multiple API formats

 

 

 

 

 

Open event server has two types of API (Application Programming Interface) formats, with one being generated by the legacy server and other one by the server side of decoupled development structure. The open event web app supported only the new version of API format, thus an error in read contents of JSON was thrown for the old version API format. To enable the support for both kind of API formats such that web app can be generated for each of them and there is no need to convert JSON files of version v1 to v2 we added an option field to the generator, where the client can choose the API version.

Excerpts and description for difference between data formats of API v1 and v2

The following excerpt is a subprogram getCopyrightData in both versions v1 and v2. The key for getting licence details in v1 is ‘licence_details’ and in v2 is ‘licence-details’. Similarly the key for getting copyright details in v1 is ‘copyright’ and in v2 is ‘event-copyright’.

So the data is extracted from the JSON files depending on the API version, the client has selected.

API V1

function getCopyrightData(event) {
 if(event.licence_details) {
   return convertLicenseToCopyright(event.licence_details, event.copyright);
 } else {
   return event.copyright;
 }
}

 

API V2

function getCopyrightData(event) {
 if(event['licence-details']) {
   return convertLicenseToCopyright(event['licence-details'], event['event-copyright']);
 } else {
   event['event-copyright'].logo = event['event-copyright']['logo-url'];
   return event['event-copyright'];
 }
}

 

Another example showing the difference between the API formats of v1 and v2 is excerpted below.

The following excerpt shows a constant ‘url’ containing the event URLs and the details. The version v1 uses event_url as a key for the main page url whereas v2 uses event-url for getting the same. A similar kind of structural difference is present for rest of the fields where the special character underscore has been replaced by hyphen and a bit of change in the name format for keys such as start_time, end_time.

API v1

const urls= {
 main_page_url: event.event_url,
 logo_url: event.logo,
 background_url: event.background_image,
 background_path: event.background_image,
 description: event.description,
 location: event.location_name,
 orgname: event.organizer_name,
 location_name: event.location_name,
};

 

API v2

const urls= {
 main_page_url: event['event-url'],
 logo_url: event['logo-url'],
 background_url: event['original-image-url'],
 background_path: event['original-image-url'],
 location: event['location-name'],
 orgname: event['organizer-name'],
 location_name: event['location-name'],
};

How we enabled support for both API formats?

To add the support for both API formats we added a options field on generator’s index page where the user chooses the type of API format for web app generation.

<label>Choose your API version</label>
<ul style="list-style-type:none">
 <li id="version1"><input name="apiVersion" type="radio" value="api_v1">   API_v1</li>
 <li id="version2"><input name="apiVersion" type="radio" value="api_v2"> API_v2</li>
</ul>

The generator depending on the version of API format, chooses the correct file where the data extraction from the input JSON files takes place. The file names are fold_v1.js and fold_v2.js for extraction of JSON v1 data and JSON v2 data respectively.

var type = req.body.apiVersion || 'api_v2';

if(type === 'api_v1') {
 fold = require(__dirname + '/fold_v1.js');
}
else {
 fold = require(__dirname + '/fold_v2.js');
}

 

The excerpts of code showing the difference between API formats of v1 and v2 are the contents of fold_v1.js and fold_v2.js files respectively.

Resources

Continue ReadingEnable web app generation for multiple API formats

Improving the JSON file upload structure – Open Event Web app

Open Event Web app generator also allows user to upload JSON file for the event data other than the API endpoint. The generator used the socket connection ID to uniquely identify uploaded files on the server which worked good for a single socket connection but failed for multiple due to overlap of connection IDs which resulted in crashing of web app. The problem is fixed by providing a unique ID to every file uploaded on the server and creating a separate field for uploaded file ID in the request body.

How to add listener for file upload?

A listener for socket ‘file’ event is added in the file app.js, which is triggered when the event namely file is emitted by the socket. The file ID kept unique by introducing a counter for number of files uploaded on the server till now and incrementing the counter subsequently for every new file.

ss(socket).on('file', function(stream, file) {
 generator.startZipUpload(count, socket);
 console.log(file);
 filename = path.join(__dirname, '..', 'uploads/connection-' +    count.toString()) + '/upload.zip';
 count += 1;
 stream.pipe(fs.createWriteStream(filename));
});

 

The procedure named startZipUpload in generator.js is executed when the zip file upload starts which further calls the helper function to make uploads directory on the server.

exports.startZipUpload = function(id, socket) {
 console.log('========================ZIP UPLOAD START\n\n');
 distHelper.makeUploadsDir(id, socket);
 distHelper.cleanUploads(id);
};

Creating uploads directory

Uploads directory is created in the root directory using the file system interfaces, the ID passed as parameter ensures that the file names do not overlap.

makeUploadsDir: function(id, socket) {
 fs.mkdirpSync(uploadsPath + '/connection-' + id.toString());
 socket.emit('uploadsId', id);
}

Embedding uploads ID with the data

After the successful creation of uploads directory and the file, the socket emits the ID of uploaded file through the event uploadsID. The value of ID thus received is embedded in the object namely data along with the other entries in the form.

socket.on('uploadsId', function(data) {
 initialValue = data;
});

function getData(initValue) {
 const data = initValue;
 const formData = $('#form').serializeArray();

 formData.forEach(function(field) {
   if (field.name === 'email') {
     data.email = field.value;
   }
   ....
   ....
   ....
   ....
 
   if (field.name === 'apiVersion') {
     data.apiVersion = field.value;
   }
 });

Resources

 

Continue ReadingImproving the JSON file upload structure – Open Event Web app

Open Event Web App – A PWA

Introduction

Progressive Web App (PWA) are web applications that are regular web pages or websites but can appear to the user like traditional applications or native mobile applications. The application type attempts to combine features offered by most modern browsers with the benefits of mobile experience. Open Event web app is a web application generator which has now introduced this new feature in its generated applications.

 

Why Progressive Web Apps?

The reasons why we enabled this functionality are that PWAs are –

  • Reliable – Load instantly and never show the downasaur, even in uncertain network conditions.
  • Fast – Respond quickly to user interactions with silky smooth animations and no janky scrolling.
  • Engaging – Feel like a natural app on the device, with an immersive user experience.

Thus where Open Event Web app generated applications are informative and only requires one time loading with functionalities like bookmarks depending on local storage of browser, we found Progressive web apps perfect to explain and demonstrate these applications as a whole.

How PWAs work?

The components associated with a progressive web application are :

Manifest: The web app manifest is a W3C specification defining a JSON-based manifest to provide developers a centralized place to put metadata associated with a web application.

Service Workers: Service Workers provide a scriptable network proxy in the web browser to manage the web/HTTP requests programmatically. The Service Workers lie between the network and device to supply the content. They are capable of using the cache mechanisms efficiently and allow error-free behavior during offline periods.

How we turned Open event Web app to a PWA?

Adding manifest.json

 {
"icons": [
    {
      "src": "./images/logo.png",
      "type": "image/png",
      "sizes": "96x96"
    }
  ],
  "start_url": "index.html",
  "scope": ".",
  "display": "standalone",
  "orientation": "portrait-primary",
  "background_color": "#fff",
  "theme_color": "#3f51b5",
  "description": "Open Event Web Application Generator",
  "dir": "ltr",
  "lang": "en-US"
}

 

Adding service workers

The initialization of service workers is done by calling an event listener namely ‘install’ :

var urlsToCache = [
 './css/bootstrap.min.css',
 './offline.html',
 './images/avatar.png'
];

self.addEventListener('install', function(event) {
 event.waitUntil(
   caches.open(CACHE_NAME).then(function(cache) {
     return cache.addAll(urlsToCache);
   })
 );
});

The service workers fetch the data from the cache when event listener ‘fetch’ is triggered. When a cache hit occurs the response data  is sent to the client from there otherwise it tries to fetch the data by making a request to the network. In case when network does not send response status code ‘200’, it sends an error response otherwise caches the data received.

self.addEventListener('fetch', function(event) {
 event.respondWith(
   caches.match(event.request).then(function(response) {
     // Cache hit - return response
     if (response) {
       return response;
     }

     var fetchRequest = event.request.clone();

     return fetch(fetchRequest)
       .then(function(response) {
         if (
           !response ||
           response.status !== 200 ||
           response.type !== 'basic'
         ) {
           return response;
         }
         var responseToCache = response.clone();
         caches.open(CACHE_NAME).then(function(cache) {
           cache.put(event.request, responseToCache);
         });
         return response;
       })
       .catch(function(err) {
         if (event.request.headers.get('Accept').indexOf('text/html') !== -1) {
           return caches.match('./offline.html');
         } else if (event.request.headers.get('Accept').indexOf('image') !== -1) {
           return caches.match('./images/avatar.png');
         } else {
           console.log(err);
         }
       });
   })
 );
});

The service workers are activated through the event listener namely ‘activate’ :

self.addEventListener('activate', function(event) {
 event.waitUntil(
   caches.keys().then(function(cacheNames) {
     return Promise.all(
       cacheNames.map(function(cacheName) {
         if (cacheName !== CACHE_NAME) {
           console.log('Deleting cache ' + cacheName);
           return caches.delete(cacheName);
         }
       })
     );
   })
 );
});

Adding service workers and manifest to the generator

Since we need to add the service workers and manifest to every web application generated through app generator, we copy the files ‘sw.js’ and ‘manifest.json’ in the directory structure of that particular web app using filestream module with the help of two abstract functions ‘copyServiceWorker’ and ‘copyManifestFile’ present in ‘distHelper.js’ code file.

 

distHelper.copyServiceWorker(appFolder, hashObj['hash'], function (err) {
 if (err) {
   console.log(err);
   logger.addLog('Error', 'Error occurred while copying service worker file', socket, err);
   return done(err);
 }
 return done(null);
});

distHelper.copyManifestFile(appFolder, eventName, function(err) {
 if (err) {
   console.log(err);
   logger.addLog('Error', 'Error occured while copying manifest file', socket, err);
   return done(err);
 }
 return done(null);
});

 

Further Improvements

Enabling push notifications for the bookmarked tracks and sessions. The user would be notified about the upcoming events through the notifications in the way the native mobile applications do.

Resources

 

Continue ReadingOpen Event Web App – A PWA