Status Badges for Repositories Registered to Yaydoc

Yaydoc, our automatic documentation generation and deployment project, generates and deploys documentation for each of its registered repository at every commit. It is possible that due to any misconfiguration in users’ project the build process may fail. Hence, it is vital for an improved user experience to store the build status for at least the most recent build process.

There are different ways with which a user can be notified about the build status after a commit. At Yaydoc, we chose to notify the user by emailing a status report. Since sending an email at each at every commit can be quite annoying, we chose to limit it to specific scenarios. We decided that we will send the mail

  • On the first build after the repository is registered to Yaydoc, irrespective of the status
  • On every failed build
  • On the change of build status (Success to Failed or vice versa)
  • To the user who registered the repository to Yaydoc
exports.updateBuildStatus = function (name, buildStatus) {
  async.waterfall([
    function (callback) {
      Repository.setBuildStatusToRepository(name, buildStatus, 
      function (error, repository) {
        callback(error, repository);
      });
    },
    function (repository, callback) {
      if (repository.mailService === true && 
          (repository.buildStatus === false || buildStatus === false || 
           repository.buildStatus === undefined)) {
        User.getUserByUsername(repository.registrant.login, 
        function (error, user)) {
          callback(null, user, repository);
        }
      }
    }
  ], function (error, user, repository) {
    mailer.sendMailOnBuild(buildStatus, user.email, repository);
  });
};

Considering the fact that the user may not wish to receive build emails and hence made them configurable by adding a mailService: Boolean  key in repository’s collection.

Taking this forward, we then decided to generate status badges similar to how Travis and other Continuous Integration platform do. Since we now store a `buildStatus` for each repository, we can use it to generate an svg image to be added to README files of repositories. We generated  the status badges using Shields.io and added them to the route /<owner>/<reponame>.svg.  The dynamicity of image generated is achieved by retrieving the value of buildStatus and render the images with different constructs based on its value.

router.get(‘/:owner/:reponame.svg’, function (req, res, next) {
  var fullName = req.params.owner + ‘/’ + req.params.reponame;
  Repository.getBuildStatusByRepositoryName(fullName, function(error, result)) {
    var buildStatus = ‘invalid’; var width =94’; 
    var color = ‘#9f9f9f’; var x =70.5’;
    
    if (result.buildStatus) {
      buildStatus = ‘success’; width =102’; color = ‘#97CA00’; x =74.5’;
    } else {
      buildStatus = ‘failed’; width =88’; color = ‘#E05d44’; x =67.5’;
    }

    res.set(‘Content-Type’, ‘image/svg+xml’);
    res.render(“badge”, {
      status: buildStatus,
      width: width,
      color: color,
      x: x,
    });
  }
});

The status tags generated can then be added as:

[![Yaydoc Status] (https://yaydoc.herokuapp.com/imujjwal96/prelimQuiz.svg)] (https://yaydoc.herokuapp.com/imujjwal96/prelimQuiz)

Resources:

  1. Shields.io: Quality metadata badges for open source projects – https://shields.io
  2. Async utilities for node and browser – https://caolan.github.io/async/
Continue Reading

Scheduling Jobs to Check Expired Access Token in Yaydoc

In Yaydoc, we use the user access token to do various tasks like pushing documentation, registering webhook and to see it’s status.The user access token is very important to us, so I decided of adding Cron job which checks whether the user token has expired or not. But then one problem was that, if we have more number of users our cron job will send thousands of request at a time so it can break the app. So, I thought of queueing the process. I used `asyc` library for queuing the job.

const github = require("./github")
const queue = require("./queue")

User = require("../model/user");

exports.checkExpiredToken = function () {
  User.count(function (error, count) {
    if (error) {
      console.log(error);
    } else {
      var page = 0;
      if (count < 10) {
        page = 1;
      } else {
        page = count / 10;
        if (page * 10 < count) {
          page = (count + 10) /10;
        }
      }
      for (var i = 0; i <= page; i++) {
        User.paginateUsers(i, 10,
        function (error, users) {
          if (error) {
            console.log(error);
          } else {
            users.forEach(function(user) {
              queue.addTokenRevokedJob(user);
            })
          }
        })
      }
    }
  })
}

In the above code I’m paginating the list of users in the database and then I’m adding each user to the queue.

var tokenRevokedQueue = async.queue(function (user, done) {
  github.retriveUser(user.token, function (error, userData) {
    if (error) {
      if (user.expired === false) {
        mailer.sendMailOnTokenFailure(user.email);
        User.updateUserById(user.id, {
          expired: true
        }, function(error, data) {
          if (error) {
            console.log(error);
          }
        });
      }
      done();
    } else {
      done();
    }
  })
}, 2);

I made this queue with the help of async queue method. In the first parameter, I’m passing the logic and in second parameter, I’m passing how many jobs can be executed asynchronously. I’m checking if the user has revoked the token or not by sending API requests to GitHub’s user API. If it gives a response ‘200’ then the token is valid otherwise it’s invalid. If the user token is invalid, I’m sending email to the user saying that “Your access token in revoked so Sign In once again to continue the service”.

Resources:

Continue Reading

Adding Github buttons to Generated Documentation with Yaydoc

Many times repository owners would want to link to their github source code, issue tracker etc. from the documentation. This would also help to direct some users to become a potential contributor to the repository. As a step towards this feature, we added the ability to add automatically generated GitHub buttons to the top of the docs with Yaydoc.

To do so we created a custom sphinx extension which makes use of http://buttons.github.io/ which is an excellent service to embed GitHub buttons to any website. The extension takes multiple config values and using them generates the `html` which it adds to the top of the internal docutils tree using a raw node.

GITHUB_BUTTON_SPEC = {
    'watch': ('eye', 'https://github.com/{user}/{repo}/subscription'),
    'star': ('star', 'https://github.com/{user}/{repo}'),
    'fork': ('repo-forked', 'https://github.com/{user}/{repo}/fork'),
    'follow': ('', 'https://github.com/{user}'),
    'issues': ('issue-opened', 'https://github.com/{user}/{repo}/issues'),
}

def get_button_tag(user, repo, btn_type, show_count, size):
    spec = GITHUB_BUTTON_SPEC[btn_type]
    icon, href = spec[0], spec[1].format(user=user, repo=repo)
    tag_fmt = '<a class="github-button" href="{href}" data-size="{size}"'
    if icon:
        tag_fmt += ' data-icon="octicon-{icon}"'
    tag_fmt += ' data-show-count="{show_count}">{text}</a>'
    return tag_fmt.format(href=href,
                          icon=icon,
                          size=size,
                          show_count=show_count,
                          text=btn_type.title())

The above snippet shows how it takes various parameters such as the user name, name of the repository, the button type which can be one of fork, issues, watch, follow and star, whether to display counts beside the buttons and whether a large button should be used. Another method named get_button_tags is used to read the various configs and call the above method with appropriate parameters to generate each button.

The extension makes use of the doctree-resolved event emitted by sphinx to hook into the internal doctree. The following snippet shows how it is done.

def on_doctree_resolved(app, doctree, docname):
    if not app.config.github_user_name or not app.config.github_repo:
        return
    buttons = nodes.raw('', get_button_tags(app.config), format='html')
    doctree.insert(0, buttons)

Finally we add the custom javascript using the add_javascript method.

app.add_javascript('https://buttons.github.io/buttons.js')

To use this with yaydoc, users would just need to add the following to their .yaydoc.yml file.

build:
  github_button:
    buttons:
      watch: true
      star: true
      issues: true
      fork: true
      follow: true
    show_count: true
    large: true

Resources

  1.  Homepage of Github:buttons – http://buttons.github.io/
  2. Sphinx extension Tutorial – http://www.sphinx-doc.org/en/stable/extdev/tutorial.html
Continue Reading

Display a List of Registered Repositories in User’s Dashboard

Yaydoc, our automatic documentation generation and deployment project was recently introduced with the dashboard for its registered users. Introducing a dashboard to Yaydoc enabled us to include a lot of new functionalities and processes associated with users and their registered repositories.

One of the main reasons of creating a dashboard for registered users was to enable them to access and edit configurations of the repositories they registered to Yaydoc. Hence, there was a need to display all of the registered repositories to users. These repositories include the public repositories owned by the users and organizations’ repositories the user has admin access to.

The organizations and users are retrieved separately in parallel using async.js. Async is a utility module which provides straight-forward, powerful functions for working with asynchronous JavaScript. The required functionality is achieved using async.parallel() method which runs the tasks of a collection of functions in parallel, without waiting until the previous function has completed.

async.parallel({
  organizations: function (callback) {
    github.retrieveOrganizations(user.token, function (error, organizations)) {
      callback(null, organizations);
    }
  },
  ownedRepositories: function (callback) {
    Repository.getRepositoriesByOwner(user.username, function (err, repositories)) {
      callback(null, repositories);
    }
  }, function (err, results) {
    res.render(‘dashboard’, {
      title: ‘Dashboard’,
      organizations: results.organizations,
      ownedRepositories: results.ownedRepositories
    });
  }
});

In order to show the organizations’ registered repositories  the github.retrieveOrganizations() function is updated, retrieving registered repositories from Yaydoc’s database. To perform the operation, we use async’s waterfall method. This method runs the tasks in series, passing the result of each function to the next function in that array. This function is needed since we are retrieving the organizations first and then for each organization, we retrieve all the repositories for each organization. To access each organization asynchronously, we use the async.forEachOf() method.

/**
 * Retrieve a list of organization the user has access to
 * Along with the organizations, also retrieve the registered repositories
 * @param accessToken: Access token of the user
 * @param callback
 */
exports.retrieveOrganizations = function (accessToken, callback) {
  async.waterfall([
    function (callback) {
      // Retrieve organizations
      ....
      organizations.push({
        id: organization.id,
        name: organization.login,
        avatar: organization.avatar_url
      });
      return callback(null, organizations);
    },
    function (organizations, callback) {
      async.forEachOf(organizations, function (value, key, callback)) {
        Repository.getRepositoriesByOwner(value.name, function (error, repositories) {
          value.repositories = repositories;
          update.push(value);
          callback()
        });
      }, function (err) {
        callback(null, update);
      }
    }
  ], function (error, result) {
    callback(error, result);
  });
};

Showing the list of repositories, we now plan to make these repositories configurable. Some of these configurations may include deleting the repository and enabling or disabling it from the build process.

Resources

  1. Async utilities for node and browser – https://caolan.github.io/async/
  2. Github’s organization API https://developer.github.com/v3/orgs/
Continue Reading

Implementing an Interface for Reading Configuration from a YAML File for Yaydoc

Yaydoc reads configuration specified in a YAML file to set various options during the build process. This allows users to customize various properties of the build process. The current implementation for this was very basic. Basically it uses a pyYAML, a yaml parser for python to read the file and convert it to a python dictionary. From the dictionary we extracted values for various properties and converting them to strings using various heuristics such as converting True to ”true”, False to ”false”, a list to comma separated string and None to an empty string. Finally, we exported variables with those values.

Recently the entire code for this was rewritten using object-oriented paradigm. The motivation for this came from the fact that the implementation lacked certain features and also required some refactoring for long term readability. In the following paragraph, I have discussed the new implementation.

Firstly a Configuration class was created which basically wraps around a dictionary and provide certain utility methods. The primary difference is that the Configuration class allows dotted key access. This means that you can use the following syntax to access nested keys.

theme = conf[‘build.theme.name’]

The class provides another method connect which is used to connect environment variables with configuration values. This method also takes a dotted key but provides an extension on top of that to handle the case when a certain option can take multiple values. For example,

option: my_option

Or,

option:
  - my_option1
  - my_option2

To indicate that a certain config is of this type, you can specify a “@” character at the end of the key. Anything after the “@” character is assumed to be an attribute of each element within the list. Let’s see an example of this whole process.

build:
  subproject:
    - url: <url1>
  source: “doc”
    - url: <url2>

Now to extract all urls from the above file, we’d need to do the following

config.connect(‘SUBPROJECT_URLS’, ‘[email protected]’)

To extract sources, we’ll also use the default parameter as the source option is optional.

config.connect(‘SUBPROJECT_SOURCES’, [email protected]’, default=’docs’)

Finally, The Configuration object also provides a getenv method which reads all connection and serializes values to string according to the previously described heuristics. It then returns a dictionary of all environment variables which must be set.

Resources

Continue Reading

Implementing Search Bar Using GitHub API In Yaydoc CI

In Yaydoc’s, documentation will be generated by typing the URL of the git repository to the input box from where user can generate documentation for any public repository, they can see the preview and if they have access, they can push the documentation to the github pages on one click. But In Yaydoc CI user can register the repository only if he has access to the specific repository, so I decided to show the list to the repository where user can select from the list but then it also has one problem that Github won’t give us all the user repository data in one api hit and then I made a search bar in which user can search repository and can register to the Yaydoc CI.

var search = function () {
  var username = $("#orgs").val().split(":")[1];
  const searchBarInput = $("#search_bar");
  const searchResultDiv = $("#search_result");

  searchResultDiv.empty();
  if (searchBarInput.val() === "") {
    searchResultDiv.append('<p class="text-center">Please enter the repository name<p>');
    return;
  }

  searchResultDiv.append('<p class="text-center">Fetching data<p>');

  $.get(`https://api.github.com/search/repositories?q=user:${username}+fork:true+${searchBarInput.val()}`, function (result) {
    searchResultDiv.empty();
    if (result.total_count === 0) {
      searchResultDiv.append(`<p class="text-center">No results found<p>`);
      styles.disableButton("btnRegister");
    } else {
      styles.disableButton("btnRegister");
      var select = '<label class="control-label" for="repositories">Repositories:</label>';
      select += '<select class="form-control" id="repositories" name="repository" required>';
      select += `<option value="">Please select</option>`;
      result.items.forEach(function (x){
        select += `<option value="${x.full_name}">${x.full_name}</option>`;
      });
      select += '</select>';
      searchResultDiv.append(select);
    }
  });
};

$(function() {
  $("#search").click(function () {
    search();
  });
});

In the above snippet I have defined search function which will get executed when user clicks the search button. The search function will get the search query from input box, if the search query is empty it’ll show the message as “Please enter repository name”, if it is not empty it’ll hit the GitHub API to fetch user repositories. If the GitHub returns empty array it’ll show “No results found”. In between searching time “Fetching data” will be shown.

$('#search_bar').on('keyup keypress', function(e) {
    var keyCode = e.keyCode || e.which;
    if (keyCode === 13) {
      search();
    }
  });

  $('#ci_register').on('keyup keypress', function(e) {
    var keyCode = e.keyCode || e.which;
    if (keyCode === 13) {
      e.preventDefault();
      return false;
    }
  });

Still we faced some problem, like on click enter button form is automatically submitting. So I’m registering event listener. In that listener I’m checking whether the key code is 13 or not. Key code 13 represent enter key, so if the key code is 13 then i’ll prevent the form from submitting. You can see the working of the search bar in the Yaydoc CI.

Resources:

Continue Reading

Specifying Configurations for Yaydoc with .yaydoc.yml

Like many continuous integration services, Yaydoc also reads configurations from a YAML file. To get started with Yaydoc the first step is to create a file named .yaydoc.yml and specify the required options. Recently the yaydoc team finalized the layout of the file. You can still expect some changes as the projects continues to grow on but they shall not be major ones. Let me give you an outline of the entire layout before describing each in detail. Overall the file is divided into four sections.

  • metadata
  • build
  • publish
  • extras

For the first three sections, their intention is pretty clear from their names. The last section extras is meant to hold settings related to integration to external services.

Following is a description of config options under the metadata section and it’s example usage.

Key Description Default
author

The author of the repository. It is used to construct the copyright text.

user/organization
projectname

The name of the project. This would be displayed on the generated documentation.

Name of the repository
version

The version of the project. This would be displayed alongside the project name

Current UTC date
debug

If true, the logs generated would be a little more verbose. Can be one of true|false.

true
inline_math

Whether inline math should be enabled. This only affects markdown documents.

false
autoindex

This section contains various settings to control the behavior of the auto generated index. Use this to customize the starting page while having the benefit of not having to specify a manual index.

metadata:
  author: FOSSASIA
  projectname: Yaydoc
  version: development
  debug: true
  inline_math: false

Following is a description of config options under the build section and it’s example usage.

Key Description Default
theme

The theme which should be used to build the documentation. The attribute name can be one of the built-in themes or any custom sphinx theme from PyPI. Note that for PyPI themes, you need to specify the distribution name and not the package name. It also provides an attribute options to control specific theme options

sphinx_fossasia_theme
source

This is the path which would be scanned for markdown and reST files. Also any static content such as images referenced from embedded html in markdown should be placed under a _static directory inside source. Note that the README would always be included in the starting page irrespective of source from the auto-generated index

docs/
logo

The path to an image to be used as logo for the project. The path specified should be relative to the source directory.

markdown_flavour

The markdown flavour which should be used to parse the markdown documents. Possible values for this are markdown, markdown_strict, markdown_phpextra, markdown_github, markdown_mmd and commonmark. Note that currently this option is only used for parsing any included documents using the mdinclude directive and it’s unlikely to change soon.

markdown_github
mock

Any python modules or packages which should be mocked. This only makes sense if the project is in python and uses autodoc has C dependencies.

autoapi

If enabled, Yaydoc will crawl your repository and try to extract API documentation. It Provides attributes for specifying the language and source path. Currently supported languages are java and python.

subproject

This section can be used to include other repositories when building the docs for the current repositories. The source attribute should be set accordingly.

github_button

This section can be used to include various Github buttons such as fork, watch, star, etc.

hidden
github_ribbon

This section can be used to include a Github ribbon linking to your github repo.

hidden
build:
  theme:
    name: sphinx_fossasia_theme
  source: docs
  logo: images/logo.svg
  markdown_flavour: markdown_github
  mock:
    - numpy
    - scipy
  autoapi:
    - language: python
      source: modules
    - language: java
  subproject:
    - url: <URL of Subproject 1>
      source: doc
    - url: <URL of subproject 2>

Following is a description of config options under the publish section and it’s example usage.

Key Description Default
ghpages

It provides a attribute url whose value is written in a CNAME file while publishing to github pages.

heroku

It provides an app_name attribute which is used as the name of the heroku app. Your docs would be deployed at <app_name>.herokuapp.com

publish:
  ghpages:
    url: yaydoc.fossasia.org
  heroku:
    app_name: yaydoc 

Following is a description of config options under the extras section and it’s example usage.

Key Description Default
swagger

This can be used to include swagger API documentation in the build. The attribute url should point to a valid swagger json file. It also accepts an additional parameter ui which for now only supports swagger.

javadoc

It takes an attribute path and can include javadocs from the repository.

extras:
  swagger:
    url: http://api.susi.ai/docs/swagger.json
    ui: swagger
  javadoc:
    path: 'src/' 

Resources

Continue Reading

Registering Organizations’ Repositories for Continuous Integration with Yaydoc

Among various features implemented in Yaydoc was the introduction of a modal in the Web Interface used for Continuous Deployment. The modal was used to register user’s repositories to Yaydoc. All the registered repositories then had their documentation updated continuously at each commit made to the repository. This functionality is achieved using Github Webhooks.

The implementation was able to perform the continuous deployment successfully. However, there was a limitation that only the public repositories owned by a user could be registered. Repositories owned by some organisations, which the user either owned or had admin access to couldn’t be registered to Yaydoc.

In order to perform this enhancement, a select tag was added which contains all the organizations the user have authorized Yaydoc to access. These organizations were received from Github’s Organization API using the user’s access token.

/**
 * Retrieve a list of organization the user has access to
 * @param accessToken: Access Token of the user
 * @param callback: Returning the list of organizations
 */
exports.retrieveOrgs = function (accessToken, callback) {
  request({
    url: ‘https://api.github.com/user/orgs’,
    headers: {
      ‘User-Agent’: ‘request’,
      ‘Authorization’: ‘token ’ + accessToken
    }
  }, function (error, response, body) {
    var organizations = [];
    var bodyJSON = JSON.parse(body);
    bodyJSON.forEach(function (organization) {
      organizations.push(organization.login);
    });
    return callback(organizations);
  });
};

On selecting a particular organization from the select tag, the list of repositories is updated. The user then inputs a query in a search input which on submitting shows a list of repositories that matches the tag. An AJAX get request is sent to Github’s Search API in order to retrieve all the repositories matching the keyword.

$(function () {
  ....
$.get(`https://api.github.com/search/repositories?q=user:${username}+fork:true+${searchBarInput.val()}`, function (result) {
    ....
    result.items.forEach(function (repository) {
      options +=<option>+ repo.full_name +</option>’;
    });
    ....
  });
  ....
});

The selected repository is then submitted to the backend where the repository is registered in Yaydoc’s database and a hook is setup to Yaydoc’s CI, as it was happening with user’s repositories. After a successful registration, every commit on the user’s or organization’s repository sends a webhook on receiving which, Yaydoc performs the documentation generation and deployment process.

Resources:

  1. Github’s Organization API: https://developer.github.com/v3/orgs/
  2. Github’s Search API: https://developer.github.com/v3/search/
  3. Simplified HTTP Request Client: https://github.com/request/request
Continue Reading

Using API Blueprint with Yaydoc

As part of extending the capability of Yaydoc to document APIs, this week we integrated API Blueprint with Yaydoc. Now we can parse apib files and add the parsed content to the generated documentation. From the official Homepage of API Blueprint,

API Blueprint is simple and accessible to everybody involved in the API lifecycle. Its syntax is concise yet expressive. With API Blueprint you can quickly design and prototype APIs to be created or document and test already deployed mission-critical APIs. It is a documentation-oriented web API description language. The API Blueprint is essentially a set of semantic assumptions laid on top of the Markdown syntax used to describe a web API.

To Integrate API Blueprint with Yaydoc, we used an sphinx extension named sphinxcontrib-apiblueprint. This extension can directly translate text in API Blueprint format into docutils nodes. The advantage with this approach as compared to using tools like aglio is that the generated html fits in nicely with the already existent theme. Though we may in future provide ability to generate html using tools like aglio if the user prefers. Adding an extension to sphinx is very easy. In the conf.py template, we added the extension to the already enabled list of extensions.

extensions += [‘sphinxcontrib.apiblueprint’]

The above extension provides a directive apiblueprint which can be then used to include apib files. The directive is very similar to the built in include directive. The difference is just that it should be only be used to include files in API Blueprint format. You can see an example below of how to use this directive.

.. apiblueprint:: <path to apib file>

Although this is enough for projects which use the ResT markup format, This cannot be used with projects using markdown as the primary markup format, since markdown doesn’t support the concept of directives. To solve this, we used the eval_rst block provided by recommonmark in Yaydoc. It allows users to embed valid ReST within markdown and recommonmark will properly parse the embedded text as ReST. Now a user can use this to use directives within markdown. You can see an example below.

```eval_rst
.. apiblueprint:: <path to apib file>
```

In order to implement this, we used the AutoStructify class provided by recommonmark. Here’s a snippet from our conf.py template. Note that this does have far reaching effects. Now users would be able to use this to add constructs like toctree in markdown which wasn’t possible before.

from recommonmark.transform import AutoStructify

def setup(app):
    app.add_config_value('recommonmark_config', {
    'enable_eval_rst': True,
    }, True)
    app.add_transform(AutoStructify)

Let’s see all of this in action. Here’s a preview of a generated documentation with API Blueprint using Yaydoc.

Resources

Continue Reading
Close Menu