Configurable Settings for Repositories Registered to Yaydoc

Yaydoc, our automatic documentation generation and deployment project, generates and deploys documentation for each of its registered repositories. These repositories registered to Yaydoc have various configurable settings which can be edited to change the behavior of the build process and other processes surrounding it. These settings include

  • Report build status via email,
  • Get build status for pull requests to the repository,
  • Specify project branches,
  • Add or remove sub-projects to a registered master project, and
  • Enable or Disable the build process
  • Delete the repository

Report Build Status via Email

At Yaydoc, user has an option to receive a mail

  • On the first build after the repository is registered to Yaydoc, irrespective of the status
  • On every failed build
  • On the change of build status (Success to Failed or vice versa)
  • To the user who registered the repository to Yaydoc

It is possible that the user may register the repository from one email-address but wishes to receive the build status in another email. Furthermore, the user may never wish to receive any email from the repository. Keeping user’s experience at a priority, we also made it configurable for the user to disable the mail service.

This feature is implemented by adding a  mailService  attribute to the Repository Schema with status: Boolean and email: String as its sub-attributes. The feature is toggled, setting the value of ‘mailService.status’ to true or false.

const Repository = mongoose.model(‘Repository’, {
  ....
  ....
  mailService: {
    status: Boolean,
    email: String,
  },
  ....
  ....
});

Pull request implementing this feature: https://github.com/fossasia/yaydoc/pull/361/files

Get build status for pull requests to the repository

We made the generation of documentation website during each Pull Request to the registered repository. Since we do not want to increase the load on our server and also due to the fact that, since testing documentation generation during a Pull Request would be rare, and the user may not want to integrate Yaydoc to Pull Requests, we made this feature configurable with ‘disabled’ as the default configuration.

A detail description of the process can be found at: Showing Pull Request Build Status in Yaydoc | FOSSASIA blog

Specify Project Branches

The documentation for a registered repository is generated at every commit made to the repository. This process happens in commits made to any of the repository’s branches except for branches that they do not have a `.yaydoc.yml` file and the `gh-pages` branch. It is seen that In many of the open sources projects hosted on Github, the development flow follows an approach in which new features and patches are made in a separate branch of the same repository before sending a PR to merge it into the master repository. In such a case, generating documentation from all the branches is an overkill and would not be expected. Hence, we let the user specify branches from which the documentation should be generated and deployed.

The existing active branches of the repository along with the registered branches are retrieved and are used to display a Bootstrap select picker to specify branches.

router.get(‘/:owner/:repository/branches’, function (req, res, next) {
  var name = req.params.owner + ‘/’ + req.params.repository;
  async.parallel({
    branches: function (callback) {
      github.getRepositoryBranches(name, function (error, branches) {
        callback(error, branches);
      });
    },
    registeredBranches: function (callback) {
      var branches = [];
      for (var branches of registeredBranches) {
        branches.push(branch.name);
      }
      callback(error, branches);
    }
  }, function (error, results) {
    res.json({
      branches: results.branches,
      registeredBranches: results.registeredBranches
    });
  });
});

Once a user specify one or more of the existing branches of the repository, documentation build occurs only from those specified repositories if they contain the `.yaydoc.yml` file.

Pull request implementing this feature: https://github.com/fossasia/yaydoc/pull/424

Add or remove sub-projects to a registered master project

Apart from the generation of documentation of a single Github repository, we also offer the user to register sub-projects for a master project. The documentation generated from these subprojects are kept at a sub route of the website with auto-generated links at the start page. The addition and deletion of these sub-projects is also made configurable from repository settings.

Pull request implementing this feature: https://github.com/fossasia/yaydoc/pull/318

Resources:

  1. Async utilities for node and browser – https://caolan.github.io/async/
  2. Mongoose Schema Documentation: http://mongoosejs.com/docs/guide.html
Continue Reading

Store Log History for Repositories Registered to Yaydoc

Yaydoc, our automatic documentation generation and deployment project, generates and deploys documentation for each of its registered repository. For every commit made to the registered repository, there is a corresponding build process running at Yaydoc. These build processes have their own logs which are stored as text files. However, until now, these commits were never visible to the user. So, if there would have been a failed build process, the user would never know the reason behind, rendering the user unable to rectify the error.

Hence, there was a need to make these logs available to users. The initial thought was to store only the latest log overriding all the previous logs for the repository. However, it was unanimously decided by the developers to store a history of logs for the repository. The main motive behind this was to enable users to compare logs between different commits.

The content from the log files created is stored in a MongoDB collection. Following is the schema defined for the build logs.

const BuildLog = mongoose.model(‘BuildLog’, mongoose.Schema({
  repository: String,  // `full_name` of the repository
  buildNumber: {       // Incrementing number for each build
    type: Number,
    default: 0,
  },
  generate: {
    data: Buffer,      // Generate Logs content
    datetime: Date,    // Date time of generate log creation
  },
  ghpages: {
    data: Buffer,      // Github Pages Logs content
    datetime: Date,    // Date time of github pages log creation
  },
}));

The repository collection is also updated, adding a builds key storing the number of times the build process was triggered for a given repository. This key is incremented on every new build and the new value is stored along with the builds as buildNumber.

The build process involves a documentation generation and a documentation deployment script. The process of incrementing the build number in the repository occurs when we store the documentation generation logs. After that, Github pages logs are stored when the documentation deployment process is completed.

Since the logs are stored in a text file at the location temp/<email>/<filename>.txt, we had to read the file using NodeJS File system. The file is read synchronously using the fs.readFileSync(filename) function and then stored in the MongoDB collection.

/**
 * Store logs created while generating docs for a given repository
 * @param name: `full_name` of the repository
 * @param filepath: file path of the generate logs
 * @param callback
 */
module.exports.storeGenerateLogs = function (name, filepath, callback) {
  Repository.incrementBuildNumber(name, function (error, repository) {
    var buildlog = new BuildLog({
      repository: name,
      buildNumber: repository.builds,
      generate: {
        data: fs.readFileSync(filepath),
        datetime: new Date()
      }
    });
    buildlog.save(function (error, repository) {
      callback(error, repository);
    });
  });
};

/**
 * Store logs created while deploying docs for a given repository
 * @param name: `full_name` of the repository
 * @param filepath: file of the ghpages deploy logs
 * @param callback
 */
module.exports.storeGithubPagesLogs = function (name, filepath, callback) {
  Repository.getRepositoryByName(name, function (error, repository) {
    if (error) {
      callback(error);
    } else {
      BuildLog.getParticularBuildLog(repository.name, repository.builds, 
      function (error, buildLog) {
        buildLog.ghpages.data = fs.readFileSync(filepath);
        buildLog.ghpages.datetime = new Date();
        buildLog.save(function (error, buildLog) {
          callback(error, buildLog);
        });
      });
    }
  });
};

The stored logs can then be retrieved at two different routes, with /:owner/:name/logs showing a list to logs generated in at most 10 builds and /:owner/:name showing the latest log. Similar to logs generated by Travis, accessing these routes doesn’t require the user to login to Yaydoc.

/**
 * Get a single repository with a log history of 10
 * @param name: `full_name` of the repository
 * @param callback
 */
module.exports.getRepositoryWithLogs = function (name, callback) {
  Repository.aggregate([
    { $match: {name: name}},
    {
      $lookup: {
        from: ‘buildLogs’,
        localField: ‘name’,
        foreignField: ‘repository’,
        as: ‘logs’
      }
    },
    { $unwind: ‘$logs’ },
    { $sort: { ‘logs.buildNumber’: -1 } },
    { $limit: 10 }
  ]).exec(function (error, results){
    callback(error, results);
  });
};

In order to retrieve a repository along with its logs, we perform an aggregation in MongoDB which is similar to a Left Join in SQL. This is the $lookup aggregation and it performs a left outer join to an unsharded collection in the same database to filter in documents from the “joined” collection for processing. A similar method is used to retrieve the latest log by setting the limit aggregation to 1.

Resources:

  1. MongoDB Aggregation Lookup: https://docs.mongodb.com/manual/reference/operator/aggregation/lookup/
  2. Mongoose Aggregate Constructor: http://mongoosejs.com/docs/api.html#index_Mongoose-Aggregate
  3. NodeJS File System: https://nodejs.org/api/fs.html#fs_fs_readfilesync_path_options
Continue Reading
Close Menu