doc generator

A few weeks back we rolled out the web interface for yaydoc. Web UI will enable user to generate the documentation with one click and users can download the zipped format of generated documentation. In Yaydoc, we now added the additional feature of deploying the generated documentation to the GITHUB pages. In order to push the generated documentation, we have to get the access token of the user. So I used passport Github’s strategy to get the access token of the users.

passport.use(new githubStrategy({
  clientID: process.env.CLIENTID,
  clientSecret: process.env.CLIENTSECRET,
  callbackURL: process.env.CALLBACKURL
}, function (accessToken, refreshToken, profile, cb) {
  profile.token = accessToken;
  cb(null, profile)
}))

passport.serializeUser(function(user, cb) {
  cb(null, user);
});

passport.deserializeUser(function(obj, cb) {
  cb(null, obj);
}); 

After setting the necessary environment variables, we have to pass the strategy to the express handler.

router.get("/github", function (req, res, next) {
  req.session.uniqueId = req.query.uniqueId;
  req.session.email = req.query.email
  req.session.gitURL = req.query.gitURL
  next()
}, passport.authenticate('github', {
  scope: [
    'public_repo',
    'read:org'
  ]
}))

For maintaining state, I’m keeping the necessary information in the session so, that in the callback URL we know which repository have to push.

router.get("/callback", passport.authenticate('github'), function (req, res, next) {
  req.session.username = req.user.username;
  req.session.token = req.user.token
  res.redirect("/deploy")
})

router.get("/deploy", function (req, res, next) {
  res.render("deploy", {
    email: req.session.email,
    gitURL: req.session.gitURL,
    uniqueId: req.session.uniqueId,
    token: crypter.encrypt(req.session.token),
    username: req.session.username
  })
})

Github will send the access token to our callback. After this I’m templating the necessary information to the jade deploy template where it’ll invoke the deploy function via sockets. Then we’ll stream all the bash output log to the website.

io.on('connection', function(socket){
  socket.on('execute', function (formData) {
    generator.executeScript(socket, formData);
  });
  socket.on('deploy', function (data) {
    ghdeploy.deployPages(socket, data);
  });
});

exports.deployPages = function (socket, data) {
  var donePercent = 0;
  var repoName = data.gitURL.split("/")[4].split(".")[0];
  var webUI = "true";
  var username = data.username
  var oauthToken = crypter.decrypt(data.encryptedToken)
  const args = [
    "-e", data.email,
    "-i", data.uniqueId,
    "-w", webUI,
    "-n", username,
    "-o", oauthToken,
    "-r", repoName
  ];
  var process = spawn("./publish_docs.sh", args);

  process.stdout.on('data', function (data) {
    console.log(data.toString());
    socket.emit('deploy-logs', {donePercent: donePercent, data: data.toString()});
    donePercent += 18;
  });

  process.stderr.on('data', function (data) {
    console.log(data.toString());
    socket.emit('err-logs', data.toString());
  });

  process.on('exit', function (code) {
    console.log('child process exited with code ' + code);
    if (code === 0) {
      socket.emit('deploy-success', {pagesURL: "https://" + data.username + ".github.io/" + repoName});
    }
  });
}

Once documentation is pushed to gh-pages, the documentation URL will get appended to the web UI.

Resources:

• https://help.github.com/articles/creating-project-pages-using-the-command-line/

In yaydoc WebUI, documentation is generated using sphnix and can be downloaded as a zip file. If the user wants to see a preview of the documentation they have to unzip the zipped file and have to check the generated website for each and every build. So, we decided to implement preview feature to show the preview of generated documentation so that user will have an idea of how the website would look. Since WebUI is made with Express, we implemented the preview feature using Express’s static function. Mostly static function is used for serving static assets but we used to serve our generated site because all the generated sites are static. All the generated documentation will have an unique id and all the unique ids are generated as per uuidv4 spec. The generated document will be saved and moved to the unique folder.

mv $BUILD_DIR/_build/html $ROOT_DIR/../${UNIQUEID}_preview && cd $_/../

var express = require("express")
var path = require("path")
var favicon = require("serve-favicon");
var logger = require("morgan");
var cookieParser = require("cookie-parser");
var bodyParser = require("body-parser");
var uuidV4 = require("uuid/v4");

var app = express();
app.use(logger("dev"));
app.use(bodyParser.json()); 
app.use(bodyParser.urlencoded({ extended: false }));
app.use(cookieParser()); app.use(express.static(path.join(__dirname, "public")));
app.use("/preview", express.static(path.join(__dirname, "temp")))

app.listen(3000)

The above snippet is the just a basic Express server. In which, we have a route /preview and with the express static handler. Pass the path of your generated website as  argument, Then your sites are served over /preview route.

Resources:

• https://expressjs.com/en/starter/static-files.html

As a developer, working on a web application, you may want your users to be able to edit a file stored in your webserver. There may be certain use cases in which this may be required. Consider, for instance, its use case in Yaydoc.

Yaydoc allows its users the feature of continuous deployment of their documentation by adding certain configurations in their .travis.yml file. It is possible for Yaydoc to achieve the editing of the Travis file from the Web App itself.

To enable the support of certain functionality in your web application, I have prepared a script using ExpressJS and Socket.IO which can perform the following action. At the client side, we define a retrieve-file event which emits a request to the server. At the server side, we handle the event by executing a retrieveContent(...) function which uses spawn method of child_process to execute a script that retrieves the content of a file.

// Client Side
$(function () {
 var socket = io();
 ....
 ....
 $(“#editor-button”).click(function () {
   socket.emit(“retrieve-file”);
 });
 ....
 ....
});

// Server Side
io.on(“connection”, function (socket) {
 socket.on(“retrieve-file”, function () {
   retrieveContent(socket);
 });
 ....
 ....
});

var retrieveContent = function (socket) {
 var process = require(“child_process”).spawn(“cat”, [“.travis.yml”]);
 process.stdout.on(“data”, function (data) {
   socket.emit(“file-content”, data);
 });
};

After the file content is retrieved from the server, we use a Javascript Editor like ACE to edit the content of the file. Making all the changes to the file, we emit a store-content event. At the server side, we handle the event by executing a storeContent(…) function which uses exec method of child_process to execute a bash script that stores the content to the same file.

$(function () {
 var socket = io();
 var editor = ace.edit(“editor”);
 ....
 ....
 socket.on(“file-content”, function (data) {
   editor.setValue(data, -1);
 });
 ....
 ....
 $(“#save-modal”).click(function () {
   socket.emit(“store-content”, editor.getValue());
 });
});

io.on(“connection”, function (socket) {
 ....
 socket.on(“store-content”, function (data) {
   storeContent(socket, data);
 });
});

var storeContent = function (socket, data) {
 var script = ‘truncate -s 0 .travis.yml && echo “’ + data + ‘“ >> .travis.yml’;
 var process = require(“child_process”).exec(script);

 process.on(“exit”, function (code) {
   if (code === 0) {
     socket.emit(“save-successful”);
   }
 });
};

After successful execution of the script, a successful event is sent to the client-side which can then be handled.

A minimal sample of this application can be found at: https://github.com/imujjwal96/socket-editor

Yaydoc takes as its input the information about a user’s repository containing the documentation in Markup files and generates a static website from it. The website also includes search functionality within the documentation. It supports various built-in and custom Sphinx themes.

Since the Web User Interface is now prepared with some solid features, it was time to deploy. We chose Heroku for this because of the ease with which we can build and scale the application at free of cost.

Yaydoc consists of two components; A Web User Interface and the Generation and Deployment Scripts. The Web UI being developed with NodeJs and the scripts involving Python modules, require us to include the following buildpacks

• heroku/nodejs
• heroku/python
• https://github.com/imujjwal96/heroku-buildpack-pandoc.git

We need to set certain Environment Variables in Heroku for proper functioning of the Yaydoc. These include

• CALLBACKURL – URL where Github must return to after successful authentication
• CLIENTID – Unique Client-Id generated by Github OAuth Application
• CLIENTSECRET – Unique Client-Secret generated by Github OAuth Application
• ENCRYPTION_KEY – Required to encrypt Personal Access Token of the user
• ON_HEROKU – True, since the application is deployed to Heroku
• PYPANDOC_PANDOC – Location of Pandoc binaries
• SECRET – A very secret token

Steps for Manual Deployment

1. Install Heroku on your local machine.

   • If you have a linux based Operating Systems, type the following command in the terminal

wget -qO- https://cli-assets.heroku.com/install-ubuntu.sh | sh

• • Details for other Operating Systems can be found at https://devcenter.heroku.com/articles/heroku-cli

• Login to Heroku Server (Create an account if it doesn’t exist)

  • Type in the following command in your server

heroku login

• • Enter your credentials and login.

• Deploy Yaydoc to Heroku

  • Clone the original yaydoc repository or your own fork

git clone https://github.com/<username>/yaydoc.git

• • Move to the directory of the cloned repository

cd yaydoc/

• • Create a Heroku application using the following command

heroku create <your-app-name>

• • Add buildpacks to the application using the following commands

heroku buildpacks:set heroku/nodejs
heroku buildpacks:add --index 2 heroku/python
heroku buildpacks:add --index 3 https://github.com/imujjwal96/heroku-buildpack-pandoc.git

• • Set Environment Variables using the following commands

heroku config:set CALLBACKURL=https://<your-app-name>.herokuapp.com/callback
heroku config:set CLIENTID=<github-generated>
heroku config:set CLIENTSECRET=<github-generated>
heroku config:set ENCRYPTION_KEY=AVERYSECRETTOKENOFSPECIFICLENGTH
heroku config:set ON_HEROKU=true
heroku config:set PYPANDOC_PANDOC=~/vendor/pandoc/bin/pandoc
heroku config:set SECRET=averysecrettoken

• • Now deploy your code

git push heroku master

• • Visit the app at the URL generated by its app name

heroku open

Yaydoc consists of two components:

1. A configuration for various Continuous Integration software including Travis CI among others.
2. A Web User Interface

Since the initial stage of its development, the team has been focused on developing a `documentation generation` script and a `publish to Github Pages` script. These scripts have been developed and tested by using Travis CI.

We are now at that stage in the development of the project that we can generate the documentation of a project and keep it updated with every push in the Github repository that consists of changes in the documentation. A sample of this can be seen at https://yaydoc.fossasia.org which is a deployment of the documentation of Yaydoc using its own scripts.

After having enough confidence in the working of the script, we have now shifted our inclination towards developing a Web User Interface for the app. The WebUI is intended to perform various functionalities. These include, among others:

• Generate the documentation and Download the static files in a compressed format.
• Generate the documentation and make them available for a Preview
• Generate the documentation and Deploy them to Heroku
• Generate the documentation and Deploy them to web server using SFTP

NOTE:- The aforementioned functionalities are not exhaustive. Also, they are not certain to be developed if they are not fruitful for the users of Yaydoc. We do not intend to bloat the application with features and functionalities that may never be used.

Technology Stack

The first issue that comes with developing any Web Application is the selection of its technology stack. With a huge number of languages and their web application frameworks, it becomes very difficult to reach a conclusion. After a lot of discussions, NodeJS was selected.

The User Interface involves various technologies including

1. NodeJS – A JavaScript runtime.
2. ExpressJS – A minimal and flexible Node.js web application framework.
3. Pug (ex – Jade) – A high-performance template engine implemented for NodeJS.
4. Socket.IO – A JavaScript library for realtime web application that enables realtime, bi-directional communication between web clients and servers.

ExpressJS is set up using the express-generator as it prepares a proper minimal architecture which makes it easy to scale up the application. Since the HTML part of the application will be minimal, Pug was chosen as it has a very clean and easy to read syntax. The use of Socket.IO became necessary as the app has a bidirectional communication with the `GENERATE` script sending its log output to the front-end.

Components of the Web User Interface

The UI consists of a Form that asks the user to input

1. Email address – To provide a unique identity for a user to isolate the documentation
2. GITURL – URL of the repository which consists the docs to be generated
3. Doc  Theme – A dropdown that consists of built in Sphinx themes.

Out of the various arguments used to generate documentation in Sphinx, following are assumed

• AUTHOR – Name of the user/organization of the repository
• PROJECTNAME – Name of the repository
• DOCPATH – Documentations are assumed to be stored at “docs/”

Apart from the form, the UI also has a block that is used to display the logs while the bash script is running in the backend.

The components defined above are those that have been developed and are being tested rigorously. Since the app is constantly being developed with new features added almost daily, new components will be added to the User Interface.