Author: championswimmer

This author has written 18 articles

Using compression middleware in Node/Express

If you’re using ExpressJS to host your server, there’s a small performance tweak that you can do (works on universally on any kind of server – HTML templates to JSON REST APIs to image servers) – which is to enable GZIP compression.

The changed required to do it should be very simple. First install the compression npm package

 npm install --save compression

In you app.js (or wherever you start your express server), modify the code to include the compression middleware like this

var express = require('express);
var app = express();

var compression = require('compression');

app.use(compression());

// Rest of your app code
// app.get (endpoint, callback) 
// app.listen(port, callback)

This should easily reduce page loads time to the order of 15-20%.

For example, page load before compression middleware added 72f7abf2-4899-11e6-9cd5-68a7addaf3a6

 

And Page load time after compression is added69e6fba8-4899-11e6-9626-a25c26ea7d2b

Continue ReadingUsing compression middleware in Node/Express

Unit testing nodejs with Mocha and Chai

There are a lot of unit testing frameworks available for Javascript, Jasmine and Karma being some of the older and more popular ones.

Jasmine and Karma, though are, originally designed for browser-side JS, and hence, frameworks like NodeUnit and Mocha have become more popular with server-size JS.
We needed code coverage reports to work after the unit tests, and the Jasmine-Node reports were not sufficient, so we just moved to using Mocha.

When using Mocha, we can use some assert library (which is not necessary, but makes life a hell lot easier). We are using chai at the open-event-webapp..

First of all install mocha globally –

npm install -g mocha

And write your tests in the test/ folder that mocha by default considers as the folder containing your test specs. For example we have our tests here – https://github.com/fossasia/open-event-webapp/tree/master/test

Writing a simple mocha test is as easy as this –

var assert = require('chai').assert;
describe('Array', function() {
  describe('#indexOf()', function() {
    it('should return -1 when the value is not present', function() {
      assert.equal(-1, [1,2,3].indexOf(5));
      assert.equal(-1, [1,2,3].indexOf(0));
    });
  });
});

The first parameter inside describe() is just to show the tests in a aesthetic way in the console.

You can see our tests described in this file  – https://github.com/fossasia/open-event-webapp/blob/master/test/serverTest.js

And attached below is an screenshot of the terminal after I have run the command mocha in the root of my project

Screenshot from 2016-07-10 04-42-26

Continue ReadingUnit testing nodejs with Mocha and Chai

Handling file uploading in HTML5 and Express

The Open Event Webapp Generator has a pure HTML front end form, and Express based backend.

 

We needed an option where users can upload their own JSON files to create a schedule page for it.

On the HTML form that’s easy. As you can see here we simple add input tags with type=”file”


<input type="file" name="sponsorfile" id="sponsorfile">

In our express app, we need to use multer to be able to handle file uploads.

We create a middleware called uploadedFiles, and pass the middleware to the get() block

 

var express = require('express');
var multer = require('multer');
var app = express();

var upload = multer({dest: 'uploads/'});

var uploadedFiles = upload.fields([
  {name: 'speakerfile', maxCount: 1},
  {name: 'sessionfile', maxCount: 1},
  {name: 'trackfile', maxCount: 1},
  {name: 'sponsorfile', maxCount: 1},
  {name: 'eventfile', maxCount: 1},
  {name: 'locationfile', maxCount: 1}
]);

app.post('/live', uploadedFiles, function(req, res) {
         // req.files has the files
         // req.body has the POST body params
  });
});

Now we can access the files inside req.files (that will have a path to the temporary location of the file, you’ll have to use some filesystem module to read the files)

Continue ReadingHandling file uploading in HTML5 and Express

Sending a streaming zip file in node.js

The webapp generator that we set up is now up and running on a heroku instance and is working to generate zips of generated websites.

One of the requirements was that when the front-end presses, the “Generate” button, a request is created to the backend node app to start creating the html pages, and pack them into a zip file. Then the said zip file will be made available to download.

I intended for the zip file to become available to download immediately and the server to continuously pack data into the zip, as the user downloads it – a process known as streaming zip serving. (If you might have noticed, Google Drive uses this too. You do not know the final zip size before the download has completed.)

If the server takes time in creating the contents of the zip, this method can help, as it will allow the user to start downloading the file, before all the contents are finalised.

 

To achieve the mentioned goal, the idea Node.js module to use archiver.

If you look at the app.js code you’ll find on the /generate POST endpoint, we take the ‘req’ params and pass to a pipeToRes() function, which is defined in generator.js

app.post('/generate', uploadedFiles, function(req, res, next) {
console.log(req.files);
console.log(req.body);
res.setHeader('Content-Type', 'application/zip');
generator.pipeZipToRes(req, res);
});

And in the generator, you can find this block of code that pipes the streaming archive to the response.

 

      const zipfile = archiver('zip');

      zipfile.on('error', (err) => {
        throw err;
      });

      zipfile.pipe(res);

      zipfile.directory(distHelper.distPath, '/').finalize();

We zip the directory defined in distPath and the archiver API has a fluent call to .finalize() that finished the download for the user, when the zip is completed.

Continue ReadingSending a streaming zip file in node.js

Webapp: The generator for making schedule pages

Why a static HTML generator

 

Before we start working on more advanced features like push notifications and iCal exports etc, we have been working on getting a generator (a simple node.js script) up, that can take data for an event either in form of json files, or from a given API in open-event data format, and generate a schedule page.

It has been used to generate the programm page of OpenTechSummit 2016 (http://2016.opentechsummit.net/programm/)

It is based on the open-event-scraper   project of FOSSASIA, and some more features had been added when developing it for OTS16. Some of the new features include –

  • Ability to define copyright and license in the API/Json, and generator adds it in the footer
  • Ability to define sponsors (support for upto 3 levels are there), and the generator adds sponsor logos with links at the bottom of the page.
  • Ability to embed audio, slides and videos into the session cards.

How the process works

Right now if you take a look at the open-event-scraper project under opentechsummit (which is a fork from the FOSSASIA repo, and being used to develop the new features) , you’ll see the process goes like this –

  1. The scraper.py file scrapes data about sessions and speakers from an internal Google Sheet we have. Then the event.py file gets data about the event itself (copyright, links to social channels).

    This step is not needed if we are using a JSON API endpoint. This is needed only if the data source is a Google Sheet, then local JSON files are created.

  2. Once we have an endpoint or JSON files locally downloaded, there a node.js script – generator.js   which generates a static HTML page.
  3. The generated HTML page is based on a handelbars template – schedule.tpl where all the required markup is there.
  4. And finally there is our own stylesheet which is called schedule.css  and is a very lightweight styling addition, on top of what is majorly a vanilla bootstrap layout.

The road ahead

Going forward we will pull back in this source code to our main repo https://github.com/fossasia/open-event-webapp .

Then we’ll add some parameters that can be fed to generator.js when calling it, like  –

  • Name of event
  • Color scheme
  • URL of endpoints

This will have a minimal form-like frontend

We can host this on heroku then, where filling the form, will run the generator.js and the generated HTML and associated CSS files will be available as a zip.

Continue ReadingWebapp: The generator for making schedule pages

How to use Open Event framework to host your event website and app

So, we intended to write “Open Event” to allow event/conference organisers to be able to have their own website, and android app without having to code or build them. Our GSoC project is not yet complete, and we are adding new feartures everyday, but it is at a stage where, if you have a small event to host in your school or something, you can take a shot at it, to get versed with how the server works and how the app and website is generated.

The “Open Event” system consists of a server (code), where organisers make an event database, and add list and details of all speakers, sessions, sponsors, locations etc.

We currently have a dev and staging server set up, where organisers can create an account and try out the interface. Neither of these servers are production servers, and their databases keep getting reset when we change code, so use them only to learn how the server dashboard works for now, not to host your final event data.

Next, the organisers might want to have a webapp (code). The webapp is just a responsive, mobile friendly website that shows the details of the events in a easily readable format for the attendees. To create your own webapp, you need to fork this repository and adjust the parameters of the config.js to represent your app.

var config = {
"title": "OpenEvent", // Title of your event.

“apiBaseGetUrl”: “http://open-event.herokuapp.com/get/api/v1/”, // Base URL of the orga-server from where data comes.

“eventId”: “3”, // The event-id of your event
.
“use_testApi”: false // Must set to false, or else uses testcase json files.
};

After changing the config.js, you can deploy the webapp on any webserver that supports hosting static files (example bitballoon.com), or you can host it via github by creating a gh-pages branch, like the example app.

Next, the organisers, can create and deploy an android app (code) for their event. There are some automated configuration options still left to be manipulated, but organisers need to change the BASE_URL variable, in the Urls.java file to point to the data server, and change the package name to their liking in build.gradle. The app can be built using the command ./gradlew build . We have a FDroid flavour, that uses Open Street Maps instead of Google Maps and is completely and purely FOSS.

Continue ReadingHow to use Open Event framework to host your event website and app

Selecting a data API

As I had already mentioned in the earlier post as a part of our GSoC project we are making a framework that lets Event/Conference organisers easily generate their own Webapps and Androi Apps for the event.

So the initial step was the concur on a data API. Because for the framework to work seamlessly for multiple events, all of them must be able the collate their data in form of a single API, so that the core components of the web/mobile app that display the data can be consistent.

Fortunately, there existed already an effort to create a standard API for event data – the re-data API made by OpenDataCity for the re:publica event.

While trying to remain compliant with the re-data API we started off with a simplified subset of it.

Our API will have semantic versioning, and initially we are establishing v1 of it. So all API endpoints have the prefix /get/api/v1/

There is one top level /event endpoint that lists the basic info of all the events. If further details of an event is needed, we need to navigate into the namespace /event/<event-id>/

As you can see listed on your API docs here there are endpoints in the form of
/event/<event-id>/sessions
/event/<event-id>/speakers
/event/<event-id>/tracks
and so on, to return a list of sessions, speakers etc for the given event-id.

Continue ReadingSelecting a data API

Open Event Management Application: An Exciting Beginning

The idea of Open Event is to build an App and Management Utility for Tech Conferences and related events. It was selected under Google Summer of Code 2015. I will be working with the organisation FOSSASIA which is the largest Open Source organisation in Asia. To add to my excitement, my mentor is @mariobehling who is none other than the founder of Lubuntu. And my other mentor is @dukeleto who has been teaching us a lot of useful stuff like how to write scrum mails, manage issue tracking and documentation on github, among others.

So, let me describe a little more about the project. We call it ‘Open Event’, and we aim to make it a easy to use solution for all Open Source (or otherwise) event/conference/seminar/workshop organisers to be able to host data about their sessions, speakers, tracks, and be able to show this to the attendees using a Webapp and an Android App. You can always read more about the project goals and track the current progress at the umbrella project on Github.

Rafal is mainly working on the server that will host the data and he’s writing that up in Python, and using flask-admin to create the dashboard. Manan is responsible for the Android app (which I secretly wish I had got, because I love to work more on Android than anything else), while I am going to create the webapp .

The idea behind both the Android and webapp will be to allow event organisers to generate their own apps, without having much technical knowledge. Ideally, when our project is finished, The organisers should be able to create an Android app and a webapp, without writing a single line of code. They would just fill the event name, the color scheme etc, and then fill the data about the sessions, speakers, sponsors, tracks, locations etc of the event.

We hope by the end of the summer we can be ready with a service/product that all event organisers would love to use to host their events on in the future. I’ll follow this blog post up soon with technical details about each component, and the API schema we have adopted.

Right now, it’s 4 days past the ‘coding start’ date, and I have not coded a lot, because I have spent most of my time trying to learn AngularJS, which was very new to me. I should get back to writing more code, and stop writing blog articles, or else Duke and Mario won’t be too happy with me 😛

Continue ReadingOpen Event Management Application: An Exciting Beginning