Advanced configurations in Yaydoc’s Web UI

Yaydoc’s User Interface consists of a form with three required fields; the user’s email address, git repository’s URL, and a theme for the generated website. Specific values of these fields are the minimum requirement to generate documentation for a project. There are certain other configuration variables for whom we assumed default values. Among these, we assumed `docs/` directory or the directory specified in the `yaydoc.yml` configuration file as the default path for the documentation. Also, `Default Branch` is assumed as the branch to generate documentation website. However, this cannot guarantee the generation of docs for every other project. These configurations can have different values based on a project.

Thus, there was a need to include certain input values for advanced configuration. The addition of these configurations in the UI doesn’t compel the user to specify them. In our attempt to improve user’s experience, we show the default values to the user when they are specifying custom values for these configurations.

If the user doesn’t specify a value for the repository’s branch, a default value is retrieved from Github’s Repository Components API, taking repository’s URL from the required input as the input URL.

/**
 * Setting the branch name with `default_branch` attriburte from
 * Github’s Repository Components API
 * @param gitUrl: URL of the github repository
 */
setDefaultBranchName: function (gitUrl) {
  var owner = gitUrl.split(“/”)[3] || ‘’;
  var repository = gitUrl.split(“/”)[4] || ‘’).split(‘.’)[0] || ‘’;
  $.get(‘https://api.github.com/repos/’ + owner + ‘/’ + repository, {
    headers: {“User-Agent”: “Yaydoc”}
  }).complete(function (data) {
    $(“#target_branch”).val(data.responseJSON.default_branch);
  });
}

There are certain cases in which the design of the Web User Interface could have been confusing. Since we are displaying all the advanced configurations at once, it could’ve appeared to the users that they are specifying empty values for the other. Thus to handle this, inputs were enabled on toggle when a checkbox beside them was checked. This was achieved making following changes in the front end of the code.

/**
 * Toggle editing of Branch Name input
 */
$(“#btnEditBranch”).click(function () {
  styles.toggleEditing(“target_branch”);
  ....
  ....
});

/**
 * Toggle Enabling/Disabling an input tag
 * @param id: `id` attribute of input tag
 */
toggleEditing: function (id) {
  const input = $(‘#’ + id);
  if (input.attr(‘disabled’)) {
    input.removeAttr(‘disabled’);
    $(‘#checkbox_’ + id).removeClass(‘glyphicon-unchecked’).addClass(‘glyphicon-check’);
  } else {
    input.attr(‘disabled’, ‘disabled’);
    $(‘checkbox_’ + id).removeClass(‘glyphicon-check’).addClass(‘glyphicon-unchecked’);
  }
}

Introducing advanced configurations to the User Interface has opened the possibility for even more projects to generate and deploy docs with much lesser constraints. One of our main aim for this project is to have a fairly simple UI and UX and we hope to bring further updated to achieve that.

Resources:

  1. Github’s Repository API: https://developer.github.com/v3/repos/
  2. jQuery’s AJAX Requests: https://api.jquery.com/jquery.get

Displaying selective logs in Yaydoc with downloadable detailed logs

Yaydoc, our automatic documentation generation and deployment project, at the crux of it consists of bash scripts. These bash scripts perform various actions, including documentation generation, deployment to Github and deployment to Heroku.

Since the inception of the User Interface of the Web UI, we have been spitting out the output of the bash script directly to the console output block without any filter. We understand that the output contains a lot of jargon that provides no essential information to the user. Hence, to improve the user experience of our platform, we decided to separate the Detailed logs and show only basic logs outlining the flow of the processes.

To implement this, we append all the logs to a unique file, sending only basic logs to stdout and stderr. Since we attend to store logs and display them in the console block of Web UI simultaneously we use the tee command, piping it with echo commands.

function print_log {
  if [ -n “$LOGFILE” ]; then
    echo -e $1 | tee -a ${LOGFILE}
  else
    echo -e $1
  fi
}

${LOGFILE}  is a unique file that has the same name as the preview directory and the compressed repository. After storing the logs in the file, the contents of the file are outputted using the cat command and is then shown to the user in a modal which is after the Detailed Logs button is clicked.

$(“#btnLogs”).click(function () {
  socket.emit(‘retrieve-detailed-logs’, data);
  ....
});

socket.on(‘retrieve-detailed-logs’, function (data) {
  var process = spawn(‘cat’, [‘temp/’+data.email+’/’+data.uniqueId+’.txt’]);
  process.stdout.setEncoding(‘utf-8’);
  process.stdout.on(‘data’, function (data)) {
    socket.emit(‘file-content’, data);
  }
});

To keep the indentation of the logs, we display the content inside the HTML pre tags. Along with displaying the detailed logs, we also let our two additional functionalities. These are ‘Copy to Clipboard’ and ‘Download as text file’. The ‘copy to clipboard’ functionality is achieved using the clipboard.js jQuery library while the `Download` functionality is achieved using res.download(file) function of ExpressJS response.

socket.on(‘file-content’, function (data) {
  new Clipboard(‘#copy-button’);
  $(‘#detailed-logs’).html(data);
  $(‘#detailed-logs-modal’).modal(‘show’);
});

Resources:

  1. https://clipboardjs.com – A modern approach to copy text to clipboard
  2. https://socket.io/ – The fastest and most reliable real-time engine

Deploying documentations generated by Yaydoc to Heroku

There are many web applications available online that generates static websites. Among these projects are two unique projects developed here at FOSSASIA. These are the Open Event WebApp Generator and Yaydoc (an automatic documentation generation and deployment project.). Since Yaydoc already supports the deployment of the generated documentations to Github pages, it was just a matter of time that the deployment to Heroku is also supported.

Heroku is an excellent cloud-based platform used as a web application deployment service. Heroku provides most of its services at free of cost to the users and is excellent to host static websites provided that a little bit of tweaking is done.

For this implementation, we use the `Platform API` provided by Heroku. Stating it’s description mentioned in the documentation,

The platform API empowers developers to automate, extend and combine Heroku with other services. You can use the platform API to programmatically create apps, provision add-ons and perform other tasks that could previously only be accomplished with Heroku toolbelt or dashboard.

In order to deploy the static websites to Heroku, we need to first prepare a bundle of source code that has been compiled and is ready for execution on the Heroku runtime. This bundle is known as a Slug.

cd temp/$EMAIL/${UNIQUE_ID}_preview
mkdir -p app
cd app

curl https://nodejs.org/dist/v6.11.0/node-v6.11.0-linux-x64.tar.gz | tar xzv > /dev/null

cp $BASE/web.js
rsync -av --progress ../ . --exclude app

cd ..
tar czfv slug.tgz ./app > /dev/null

We are using the files generated for preview to bundle them in a slug. Also, we download the NodeJS runtime files since we are deploying a static website to Heroku. Along with the static files, we require bundling a NodeJS server file (web.js) that will be used to reference the static files in the application.

After preparing the Slug, we publish the static web application to Heroku. For this, we start by creating a Heroku app using the command `heroku create <app-name>`. The app name is decided by the user when he or she fills the form in the Yaydoc Web App. Following that, we request Heroku to allocate a new slug for your app. After that, we upload the slug tar file to the platform.

# Create Heroku 
heroku create $APP_NAME

# Allocating new Slug
Arr=($(curl -u “:$API_KEY” -X \
-H ‘Content-Type:application/json’ \
-H ‘Accept: application/vnd.heroku+json;version=3’ \
-d ‘{“process_types”:{“web”:”node-v6.11.0-linux-x64/bin/node web.js”}}’ \
-n https://api.heroku.com/apps/${APP_NAME}/slugs | \
python -c “import sys,json; obj=json.load(sys.stdin);
print(obj[‘blob’][‘url’] + ‘\n’ + obj[‘id’])”))

# Upload the slug tar file
curl -X PUT \
-H “Content-Type:”\
--data-binary @slug.tgz \
“${Arr[0]}”

After uploading the slug to Heroku, we need to release the app. This is done using the following command.

curl -u “:$API_KEY” -X POST \
-H “Accept: application/vnd.heroku+json; version=3” \
-H “Content-Type: application/json” \
-d ‘{“slug”:”’${Arr[1]}’”}’ \
-n https://api.heroku.com/apps/$APP_NAME/releases

Releasing the application completes the process of deployment, making the documentation generated by Yaydoc up and running at the following URL: https://<app-name>.herokuapp.com/

 

Editing a file stored in the webserver from the Yaydoc Web App

As a developer, working on a web application, you may want your users to be able to edit a file stored in your webserver. There may be certain use cases in which this may be required. Consider, for instance, its use case in Yaydoc.

Yaydoc allows its users the feature of continuous deployment of their documentation by adding certain configurations in their .travis.yml file. It is possible for Yaydoc to achieve the editing of the Travis file from the Web App itself.

To enable the support of certain functionality in your web application, I have prepared a script using ExpressJS and Socket.IO which can perform the following action. At the client side, we define a retrieve-file event which emits a request to the server. At the server side, we handle the event by executing a retrieveContent(...) function which uses spawn method of child_process to execute a script that retrieves the content of a file.

// Client Side
$(function () {
 var socket = io();
 ....
 ....
 $(“#editor-button”).click(function () {
   socket.emit(“retrieve-file”);
 });
 ....
 ....
});

// Server Side
io.on(“connection”, function (socket) {
 socket.on(“retrieve-file”, function () {
   retrieveContent(socket);
 });
 ....
 ....
});

var retrieveContent = function (socket) {
 var process = require(“child_process”).spawn(“cat”, [“.travis.yml”]);
 process.stdout.on(“data”, function (data) {
   socket.emit(“file-content”, data);
 });
};

After the file content is retrieved from the server, we use a Javascript Editor like ACE to edit the content of the file. Making all the changes to the file, we emit a store-content event. At the server side, we handle the event by executing a storeContent(…) function which uses exec method of child_process to execute a bash script that stores the content to the same file.

$(function () {
 var socket = io();
 var editor = ace.edit(“editor”);
 ....
 ....
 socket.on(“file-content”, function (data) {
   editor.setValue(data, -1);
 });
 ....
 ....
 $(“#save-modal”).click(function () {
   socket.emit(“store-content”, editor.getValue());
 });
});

io.on(“connection”, function (socket) {
 ....
 socket.on(“store-content”, function (data) {
   storeContent(socket, data);
 });
});

var storeContent = function (socket, data) {
 var script = ‘truncate -s 0 .travis.yml && echo “’ + data + ‘“ >> .travis.yml’;
 var process = require(“child_process”).exec(script);

 process.on(“exit”, function (code) {
   if (code === 0) {
     socket.emit(“save-successful”);
   }
 });
};

After successful execution of the script, a successful event is sent to the client-side which can then be handled.

A minimal sample of this application can be found at: https://github.com/imujjwal96/socket-editor

Setting up Yaydoc on Heroku

Yaydoc takes as its input the information about a user’s repository containing the documentation in Markup files and generates a static website from it. The website also includes search functionality within the documentation. It supports various built-in and custom Sphinx themes.

Since the Web User Interface is now prepared with some solid features, it was time to deploy. We chose Heroku for this because of the ease with which we can build and scale the application at free of cost.

Yaydoc consists of two components; A Web User Interface and the Generation and Deployment Scripts. The Web UI being developed with NodeJs and the scripts involving Python modules, require us to include the following buildpacks

  • heroku/nodejs
  • heroku/python
  • https://github.com/imujjwal96/heroku-buildpack-pandoc.git

We need to set certain Environment Variables in Heroku for proper functioning of the Yaydoc. These include

  • CALLBACKURL – URL where Github must return to after successful authentication
  • CLIENTID – Unique Client-Id generated by Github OAuth Application
  • CLIENTSECRET – Unique Client-Secret generated by Github OAuth Application
  • ENCRYPTION_KEY – Required to encrypt Personal Access Token of the user
  • ON_HEROKU – True, since the application is deployed to Heroku
  • PYPANDOC_PANDOC – Location of Pandoc binaries
  • SECRET – A very secret token

Steps for Manual Deployment

  1. Install Heroku on your local machine.

    • If you have a linux based Operating Systems, type the following command in the terminal
wget -qO- https://cli-assets.heroku.com/install-ubuntu.sh | sh
heroku login
    • Enter your credentials and login.
  • Deploy Yaydoc to Heroku

    • Clone the original yaydoc repository or your own fork
git clone https://github.com/<username>/yaydoc.git
    • Move to the directory of the cloned repository
cd yaydoc/
    • Create a Heroku application using the following command
heroku create <your-app-name>
    • Add buildpacks to the application using the following commands
heroku buildpacks:set heroku/nodejs
heroku buildpacks:add --index 2 heroku/python
heroku buildpacks:add --index 3 https://github.com/imujjwal96/heroku-buildpack-pandoc.git
    • Set Environment Variables using the following commands
heroku config:set CALLBACKURL=https://<your-app-name>.herokuapp.com/callback
heroku config:set CLIENTID=<github-generated>
heroku config:set CLIENTSECRET=<github-generated>
heroku config:set ENCRYPTION_KEY=AVERYSECRETTOKENOFSPECIFICLENGTH
heroku config:set ON_HEROKU=true
heroku config:set PYPANDOC_PANDOC=~/vendor/pandoc/bin/pandoc
heroku config:set SECRET=averysecrettoken
    • Now deploy your code
git push heroku master
    • Visit the app at the URL generated by its app name
heroku open

Web User Interface for Yaydoc

Yaydoc consists of two components:

  1. A configuration for various Continuous Integration software including Travis CI among others.
  2. A Web User Interface

Since the initial stage of its development, the team has been focused on developing a `documentation generation` script and a `publish to Github Pages` script. These scripts have been developed and tested by using Travis CI.

We are now at that stage in the development of the project that we can generate the documentation of a project and keep it updated with every push in the Github repository that consists of changes in the documentation. A sample of this can be seen at https://yaydoc.fossasia.org which is a deployment of the documentation of Yaydoc using its own scripts.

After having enough confidence in the working of the script, we have now shifted our inclination towards developing a Web User Interface for the app. The WebUI is intended to perform various functionalities. These include, among others:

  • Generate the documentation and Download the static files in a compressed format.
  • Generate the documentation and make them available for a Preview
  • Generate the documentation and Deploy them to Heroku
  • Generate the documentation and Deploy them to web server using SFTP

NOTE:- The aforementioned functionalities are not exhaustive. Also, they are not certain to be developed if they are not fruitful for the users of Yaydoc. We do not intend to bloat the application with features and functionalities that may never be used.

Technology Stack

The first issue that comes with developing any Web Application is the selection of its technology stack. With a huge number of languages and their web application frameworks, it becomes very difficult to reach a conclusion. After a lot of discussions, NodeJS was selected.

The User Interface involves various technologies including

  1. NodeJS – A JavaScript runtime.
  2. ExpressJS – A minimal and flexible Node.js web application framework.
  3. Pug (ex – Jade) – A high-performance template engine implemented for NodeJS.
  4. Socket.IO – A JavaScript library for realtime web application that enables realtime, bi-directional communication between web clients and servers.

ExpressJS is set up using the express-generator as it prepares a proper minimal architecture which makes it easy to scale up the application. Since the HTML part of the application will be minimal, Pug was chosen as it has a very clean and easy to read syntax. The use of Socket.IO became necessary as the app has a bidirectional communication with the `GENERATE` script sending its log output to the front-end.

Components of the Web User Interface

The UI consists of a Form that asks the user to input

  1. Email address – To provide a unique identity for a user to isolate the documentation
  2. GITURL – URL of the repository which consists the docs to be generated
  3. Doc  Theme – A dropdown that consists of built in Sphinx themes.

Out of the various arguments used to generate documentation in Sphinx, following are assumed

  • AUTHOR – Name of the user/organization of the repository
  • PROJECTNAME – Name of the repository
  • DOCPATH – Documentations are assumed to be stored at “docs/”

Apart from the form, the UI also has a block that is used to display the logs while the bash script is running in the backend.

The components defined above are those that have been developed and are being tested rigorously. Since the app is constantly being developed with new features added almost daily, new components will be added to the User Interface.

Pipelining Bash Script’s output to Webapp using Socket.io

Yaydoc, our automatic documentation generator, among other components, consists of a Web User Interface. This UI has a form that takes as its input certain information about a user’s project and generates documentations using this information in the backend with the help of a Bash Script. The caveat of executing such a Bash Script is that a user will have to wait for the processing to complete in order to get any output on the WebApp. This creates some problem as the user may not know if the process is executing properly. Furthermore, servers that are used to deploy such web applications have a limited time span within which it must send a response to a received GET or POST request. Since executing scripts may take some time, the process may lead to a Request Timeout.

We faced a similar problem with Yaydoc while deploying it to Heroku. Since Heroku has a timeout at 30 seconds, executing the Documentation Generation script lead to a Request Timeout as it takes more than 30 seconds for the execution. After doing a bit of research, we were introduced with Socket.io. Socket.IO is one of the most powerful Javascript frameworks which enables real-time bidirectional event-based communication.

At the client side, we define an “execute” event which emits the form data when the “Generate Docs” button is clicked. At the server side, we handle the event by executing a generator.executeScript(...) function with the socket and formData as its arguments.

/**
 * Client-side Event Handling
 */

$(function () {
 var socket = io();
 $(“#btnGenerate”).click(function () {
   var formData = getData();
   socket.emit(“execute”, formData);
 });
 ...
 ...
 ...
});

/**
 * Server-side Event Handling
 */
io.on(“connection”, function (socket) {
 socket.on(“execute”, function (formData) {
   generator.executeScript(socket, formData);
 });
});

 

Bash scripts are executed in NodeJS by creating child processes using the `child_process` module. This module provides four different methods for executing external applications. They are:

  1. execFile
  2. exec
  3. spawn
  4. fork

Out of these, the exec() and execFile() methods returns buffered data when the script executes successfully. We cannot use them as a solution because we need to continuously receive certain response from the server after execution of a limited number of commands in the script. Thus, we opt for spawn() which returns a stream based object every time the script produces some data. The spawn method is called in the executeScript method.

exports.executeScript = function (socket, formData) {
 ...
 ...
 var process = spawn(“./generate.sh”, args);
 process.stdout.on(“data”, function (data) {
   socket.emit(“logs”, {data: data});
 });
 ...
 ...
};

The emitted logs are then received at the client-side for display in the web application.

/**
 * Client-side Event Handling
 */
$(function () {
 ...
 ...
 socket.on(“logs”, function (data) {
   $(“#messages”).append($(“<li>”).text(data.data));
 }
 ...
 ...
});

A minimal sample of this application can be found at: https://github.com/imujjwal96/socket-bashing

Automatic deployment of Github repositories to a web server using SFTP

Git and Github are amazing tools for managing projects and are being used widely to manage various web applications among other projects.  Even though it is very efficient to manage codebase of the project, there is one concern. Since generally the project is developed at a different location and then deployed to the production server, it becomes burdensome to keep both the instances in sync.

One of the most basic methods to achieve this is to push the changes at the developing instance to Github and then pull them to the instance on the server. A slightly better and less stringent option would be to use tools like git-ftp.

From git-ftp’s documentation

If you use Git and you need to upload your files to an FTP server, Git-ftp can save you some time and bandwidth by uploading only those files that changed since the last upload.

Despite the fact that it decreases the strenuous work of accessing the server and pulling the changes, but there is a caveat. In the world of Version Control, a lot of projects are developed in collaboration with various contributors and it wouldn’t be wise to give everybody access to the server.

Among the different ways to facilitate the task of keeping the two instances in sync, there is one way using which we can automate the deployment process of a Github repository to a web server and keep it in sync. We can achieve this by using a script that executes git-ftp commands in every CI build.

The script does include some dependencies but at the crux of it are the following commands.

git config git-ftp.url $FTP_URL
git config git-ftp.user $FTP_USER
git config git-ftp.password $FTP_PASSWORD

if ! git ftp push ; then
  git ftp init
fi

Travis configuration

sudo: required

before_script:
- sudo apt-get install build-essential debhelper libssh2-1-dev
- sudo apt-get source libcurl13
- sudo apt-get build-dep libcurl13
- sudo add-apt-repository -y ppa:git-ftp/ppa
- sudo apt-get update
- sudo apt-get install git-ftp

script:
- source <(curl -s https://raw.githubusercontent.com/imujjwal96/sftp-audodeploy/master/init.sh)

Handle sequential execution of scripts in Travis CI

Many projects on Github use Travis to automatically execute certain scripts on every build. Among these projects is Yaydoc, an Automated Documentation Generation and Deployment Project. At the crux of Yaydoc are scripts that generate and deploy documentations. It uses Travis to execute these scripts on every build to keep the generated website in sync with the documentation in the markup files.

It is possible that due to some issues, the scripts may fail to execute. Unfortunately, Travis build does not fail fast. What this means is that Travis continues the build even after it encounters errors in the building process causing it to fail.

There are many projects that involve execution of multiple scripts sequentially with each dependent on the proper execution of the previous script. This requires that if one of the scripts fails then the process should stop there and none of the scripts following it should execute. Failing to achieve this can lead to some unprecedented outcomes.

One solution for this would be to handle those statements in all the scripts that could lead to a failure in the build process. Opting this approach could be burdensome as there can be multiple scripts with a huge number of commands. Also, it is hard to realise which command could fail to execute. Instead of opting for this, our approach is to use ‘Build Stages’ offered in Travis CI.

Travis’ build stages

Travis offers `build stages`, which is a way to group jobs, and run jobs in each stage in parallel, but run one stage after another sequentially. Put simply, `Build Stages` allows us make one job run only if several other, parallel jobs have completed successfully.

These build stages can be used to execute one script at each stage, with Travis exiting at the stage in which the errored script is executed.

Consider the Travis configuration defined above. This configuration describes the three stages that are involved in a real-world project, Yaydoc, which is used to Automatically Generate and Deploy the Documentation to Github Pages. It is clear from the configuration block that the three critical stages involved in the process of generating documentations using Yaydoc are

  1. Installing and Setting Up Virtual Environment
  2. Generation of Documentation
  3. Publishing Documentation

It would not be wise for the system to publish documentations that are not generated properly. Hence, these three scripts are critical, with the execution of each script dependent on the successful execution of the previous script. Each script is defined in a separate stage and thus a failed ‘Generate documentation’ script stops the build. If the above scripts were to execute normally, the ‘Publish documentation’ script would have executed even after the `Generate documentation’ script fails.

Link one repository’s subdirectory to another

Loklak, a distributed social media message search server, generates its documentation automatically using Continuous Integration. It generates the documentation in the gh-pages branch of its repository. From there, the documentation is linked and updated in the gh-pages branch of dev.loklak.org repository.

The method with which Loklak links the documentations from their respective subprojects to the dev.loklak.org repository is by using `subtrees`. Stating the git documentation,

“Subtrees allow subprojects to be included within a subdirectory of the main project, optionally including the subproject’s entire history. A subtree is just a subdirectory that can be committed to, branched, and merged along with your project in any way you want.”

What this means is that with the help of subtrees, we can link any project to our project keeping it locally in a subdirectory of our main repository. This is done in such a way that the local content of that project can easily be synced with the original project through a simple git command.

Representation:

A representation of how Loklak currently uses subtrees to link the repositories is as follows.

git clone --quiet --branch=gh-pages [email protected]/dev.loklak.org.git central-docs
cd central docs

git subtree pull --prefix=server https://github.com/loklak/loklak_server.git gh-pages --squash -m "Update server subtree"

git subtree split:

The problem with the way documentation is generated currently is that the method is very bloated. The markup of the documentation is first compiled in each sub-projects’s repository and then aggregated in the dev.loklak.org repository. What we intend to do now is to keep all the markup documentation in the docs folder of the respective repositories and then create subtrees from these folders to a master repository. From the master repository, we then intend to compile the markups to generate the HTML pages.

Intended Implementation:

To implement this we will be making use of subtree split command.

    1. Run the following command in the subprojects repository. It creates a subtree from the docs directory and moves it to a new branch documentation.
git subtree --prefix=docs/ split -b documentation
    1. Clone the master repository. (This is the repository we intend to link the subdirectory with.)
git clone --quiet --branch=master [email protected]:loklak/dev.loklak.org.git loklak_docs
cd loklak_docs
    1. Retrieve the latest content from the subproject.
      • During the first run.
git subtree add --prefix=raw/server ../loklak_server documentation --squash -m "Update server subtree"
      • During following runs.
git subtree pull --prefix=raw/server ../loklak_server documentation --squash -m "Update server subtree"
    1. Push changes.
git push -fq origin master > /dev/null 2>&1