Rendering Open Event Server’s API-Blueprint document

After writing the FOSSASIA‘s Open Event Server project API- Blueprint Document manually, we wanted to know how we could render the document, how to check it in an HTML-client friendly format and how to make it change the look as we go. In order to do that, we found two rendering ways.

They are:

1) The apiary editor:

This editor helps us to render API blueprints and print them in user readable API documented format. When we create the API blueprint manually, we always follow the pattern write an api blueprint i.e the name and metadata, then followed by resource groups and actions, which was already discussed in the last blog. In order to use the apiary editor, we start off by creating our first project. Initially during the our first use of this editor, we will get a default “polls and vote” example api project. This is a template we can use as guide. The pole/vote api looks something like this in the editor mode:

 

Apiary has a facility to test an API, document an API, inspect an API or simply edit an API. We first start off by creating a project “open-event-api”. Next, in the editor mode of the apiary, we add the contents of our api-blueprint documents.
Here is an example of how USERS API is rendered. If we get our request and response correctly, on clicking List All Users we will get a good 200 response like this in the editor:

However, if we tend to go off format with the api-blueprint, we get an invalid error:

The final rendering and how the API result can be seen in the document mode with the respective API’s request and response.
The document mode request and response look like this:

This rendered doc can be viewed publicly with the link got in the document mode. Similarly, we test it out in the editor for the rest of the ap. This is a simple way to render your api blueprint.

2)  The aglio renderer:

Since API blueprint is presented in the form of .apib format, the con side of it is it is not easily viewable by viewers. Even though we use apiary, view the rendered docs along with getting a shareable link, we would surely like the docs for our API server to be hosted in our server as well. So, we use Aglio exactly to do that .

It is an API Blueprint renderer which supports multiple themes. It converts the apib file into user readable formats such as pdf, html, etc. Here since we want to host it as a webpage, we render it in the form of .html.  It outputs static HTML of the result and can be served by any web host. Since API Blueprint is a Markdown-based document format, this lets us write API descriptions and documentation in a simple and straightforward way.
An example of how aglio rendered document in a three column format looks like:

The best thing about Aglio is not only does it support a lot many theme and templates, but it also allows you to provide your own custom theme and template to render the html file from the api blueprint.

How to use aglio renderer:

  • We first follow up with installation:
npm install -g aglio
  • After installation, we go to the folder the .apib file is stored and generate the HTML. There are 5 built in themes available with two column and three column layout. They are:
# Default theme
aglio -i input.apib -o output.html

-> This command takes as input the input.apib file as API Blueprint and creates a rendered output file named output.html.

 

# Use three-column layout
aglio -i input.apib --theme-template triple -o output.html

-> This command takes as input the input.apib file as API Blueprint and creates a rendered output file named output.html. However it uses the theme-template flag. The theme-template flag is used to define whether the layout of the rendered html is two column or three column. In this command, it is set as triple which means three column.

# Built-in color scheme
aglio --theme-variables slate -i input.apib -o output.html

-> Aglio has different color schemes that you can use while rendering the docs html file. Some of them are Olio, Streak, Slate, etc.

# Customize a built-in style
aglio --theme-style default --theme-style ./my-style.less -i input.apib -o output.html

-> Suppose you want to provide a syntactical style sheet such as SASS, LESS, etc. so as to define your own styling. You can do that as given in the above example. The my-style.less is a user defined syntactical stylesheet. This is then used to provide styling for the output file rendered.

# Custom layout template
aglio --theme-template /path/to/template.jade -i input.apib -o output.html

-> You can write your own custom layout template in a template.jade file and use that for generating the output.html instead of two or three column layout.

We run the build-in color scheme: aglio –theme-variables slate -i api_blueprint.apib -o output.html to generate our Open Event Server api document which we have something like this:

You can visit the live version of FOSSASIA‘s Open Event Server API Document right here: https://api.eventyay.com/

Continue ReadingRendering Open Event Server’s API-Blueprint document

Generating responsive email using mjml in Yaydoc

In Yaydoc, an email with a download, preview and deploy link will be sent to the user after documentation is generated. But then initially, Yaydoc was sending email in plain text without any styling, so I decided to make an attractive HTML email template for it. The problem with HTML email is adding custom CSS and making it responsive, because the emails will be seen on various devices like mobile, tablet and desktops. When going through the GitHub trending list, I came across mjml and was totally stunned by it’s capabilities. Mjml is a responsive email generation framework which is built using React (popular front-end framework maintained by Facebook)

Install mjml to your system using npm.

npm init -y && npm install mjml

Then add mjml to your path

export PATH="$PATH:./node_modules/.bin”

Mjml has a lot of react components pre-built for creating the responsive email. For example mj-text, mj-image, mj-section etc…

Here I’m sharing the snippet used for generating email in Yaydoc.

<mjml>
  <mj-head>
    <mj-attributes>
      <mj-all padding="0" />
      <mj-class name="preheader" color="#CB202D" font-size="11px" font-family="Ubuntu, Helvetica, Arial, sans-serif" padding="0" />
    </mj-attributes>
    <mj-style inline="inline">
      a { text-decoration: none; color: inherit; }
 
    </mj-style>
  </mj-head>
  <mj-body>
    <mj-container background-color="#ffffff">
 
      <mj-section background-color="#CB202D" padding="10px 0">
        <mj-column>
          <mj-text align="center" color="#ffffff" font-size="20px" font-family="Lato, Helvetica, Arial, sans-serif" padding="18px 0px">Hey! Your documentation generated successfully<i class="fa fa-address-book-o" aria-hidden="true"></i>
 
          </mj-text>
        </mj-column>
      </mj-section>
      <mj-section background-color="#ffffff" padding="20px 0">
        <mj-column>
          <mj-image src="http://res.cloudinary.com/template-gdg/image/upload/v1498552339/play_cuqe89.png" width="85px" padding="0 25px">
</mj-image>
 
          <mj-text align="center" color="#EC652D" font-size="20px" font-family="Lato, Helvetica, Arial, sans-serif" vertical-align="top" padding="20px 25px">
            <strong><a>Preview it</a></strong>
            <br />
          </mj-text>
        </mj-column>
        <mj-column>
          <mj-image src="http://res.cloudinary.com/template-gdg/image/upload/v1498552331/download_ktlqee.png" width="100px" padding="0 25px" >
        </mj-image>
          <mj-text align="center" color="#EC652D" font-size="20px" font-family="Lato, Helvetica, Arial, sans-serif" vertical-align="top" padding="20px 25px">
            <strong><a>Download it</a></strong>
            <br />
          </mj-text>
        </mj-column>
        <mj-column>
          <mj-image src="http://res.cloudinary.com/template-gdg/image/upload/v1498552325/deploy_yy3oqw.png" width="100px" padding="0px 25px" >
        </mj-image>
          <mj-text align="center" color="#EC652D" font-size="20px" font-family="Lato, Helvetica, Arial, sans-serif" vertical-align="top" padding="20px 25px">
 
            <strong><a>Deploy it</a></strong>
            <br />
          </mj-text>
        </mj-column>
      </mj-section>
      <mj-section background-color="#333333" padding="10px">
        <mj-column>
        <mj-text align="center" color="#ffffff" font-size="20px" font-family="Lato, Helvetica, Arial, sans-serif" padding="18px 0px">Thanks for using Yaydoc<i class="fa fa-address-book-o" aria-hidden="true"></i>
        </mj-column>
        </mj-text>
      </mj-section>
    </mj-container>
  </mj-body>
</mjml>

The main goal of this example is to make a responsive email which looks like the image given below. So, In mj-head tag, I have imported all the necessary fonts using the mj-class tag and wrote my custom CSS in mj-style. Then I made a container with one row and one column using mj-container, mj-section and mj-column tag and changed the container background color to #CB202D using background-color attribute, then In that container I wrote a heading which says `Hey! Your documentation generated successfully`  with mj-text tag, Then you will get the red background top bar with the success message. Then moving on to the second part, I made a container with three columns and added one image to each column using mj-image tag by specifying image URL as src attribute, added the corresponding text below the mj-image tag using the mj-text tag. At last,  I  made one more container as the first one with different message saying `Thanks for using yaydoc`  with background color #333333

At last, transpile your mjml code to HTML by executing the following command.

mjml -r index.mjml -o index.html

Rendered Email
Resources:

Continue ReadingGenerating responsive email using mjml in Yaydoc

Testing child process using Mocha in Yaydoc

Mocha is a javascript testing framework. It can be used in both nodeJS and browser as well, also it is one of the most popular testing framework available out there. Mocha is widely used for the Behavior Driven Development (BDD). In yaydoc, we are using mocha to test our web UI. One of the main task in yaydoc is documentation generation. We build a bash script to do our documentation generation. We run the bash script using node’s child_process module, but then in order to run the test you have to execute the child process before test execution. This can be achieved by mochas’s before hook. Install mocha in to your system

npm install -g mocha

Here is the test case which i wrote in yaydoc test file.

const assert = require('assert')
const spawn = require('child_process').spawn
const uuidV4 = require("uuid/v4")
describe('WebUi Generator', () => {
  let uniqueId = uuidV4()
  let email = 'fossasia@gmail.com'
  let args = [
    "-g", "https://github.com/fossasia/yaydoc.git",
    "-t", "alabaster",
    "-m", email,
    "-u", uniqueId,
    "-w", "true"
  ]
  let exitCode

  before((done) => {
    let process = spawn('./generate.sh', args)
    process.on('exit', (code) => {
      exitCode = code
      done()
    })
  })
  it('exit code should be zero', () => {
    assert.equal(exitCode, 0)
  })
 })

Describe() function is used to describe our test case. In our scenario we’re testing the generate script so we write as webui generator. As I mentioned above we have to run our child_process in before hook. It() function is the place where we write our test case. If the test case fails, an error will be thrown. We use the assert module from mocha to do the assertion. You can see our assertion in first it()  block for checking exit code is zero or not.

mocha test.js --timeout 1500000

Since documentation takes time so we have to mention time out while running mocha. If your test case passes successfully, you will get output similar to this.

WebUi Generator
    ✓ exit code should be zero

Resources:

 

Continue ReadingTesting child process using Mocha in Yaydoc

Landing Page for Phimpme Android

Landing page for any app is very important for its distribution. Its provide all the relevant informations for the app, its download/ source link and screenshot of the app for preview of its view and features. As our Phimpme app is now reached to a milestone, so we decided to bring out a landing for it.

Landing page is a simple static website which gives relevant informations and their links on a page. To develop a landing page, we can use multiple frameworks available for it. For example using Bootstrap, basic pages using html and css. As GitHub provides a free hosting service such as Github pages. I decided to work on jekyll which is easy to customize and hosted with GitHub pages.

How I did in Phimpme

  • Search for Open Source theme

There are various open source themes present for jekyll, mainly you will get for blogs. I got this awesome app landing page. It fulfils our requirements of showing a screenshot, brief description and links.

  • Create separate branch for Landing page

The GitHub pages works from a separate branch. So I firstly created a orphan branch in main Phimpme Repository. And Clear the working directory. Orphan branches are those branch which do not have any commit history. It is highly required that branch should not have any commit history, because that will irrelevant for gh-pages branch.

git checkout --orphan gh-pages
git rm --cached -r .
  • Test on Fork repo

Now rebase your branch with the gh-pages branch created on the main repository. Add the complete jekyll code here. To reflect changes on github.io page. You need to enable that in settings.

  • Enable gh-pages from settings

Go to the Settings of your repo

Scroll down to gh-pages section

Here things are clear, as we need to select the source branch from where the gh-pages is hosted. We can directly choose some open source themes built over jekyll. Also there is a option to add custom domain, such phimp.me in our case.

This all I’m writing as per the Developer perspective, where the developer have limited access to the repository and can’t access the settings of main repo. In that case push your code to by selecting the gh-pages base branch.

  • Problems I faced

The usual problem any Developer here faces is creating a orphan branch in main repo. This required removing all the files as well. Also to face less problem I recommend using some other editor such as Sublime Text, Atom etc. Or any other IDE other than Android Studio. Because in Android Studio after changing branch, it sync and rebuild again which actually takes lots of time.

Resources

 

Continue ReadingLanding Page for Phimpme Android

Documenting APIs with Yaydoc

API Documentation is a quick and concise way to tell a user about how to use a library or work with a program. It details classes, functions, parameters, return types and more. Courtesy of Sphinx, Yaydoc had build in support for Documenting APIs for Python based projects right from it’s inception. Sphinx has a built in tool autodoc which provides certain directives such as autoclass, automodule, etc which can be used to automatically extract docstrings from all specified Python packages and modules and use it to generate API documentation. As a user of Yaydoc you could add ReST sources files with appropriate directives provided by autodoc and we would handle the rest. As part of enhancing this feature we wanted to do three things.

  • Enhance support for Python
  • Extend API documentation to other languages apart from Python
  • Automate the process of generating ReST source files

For Enhancing support for python projects, we implemented a few things.

Since autodoc imports the modules it needs to document, There could be import errors if a dependency was not met. To fix this issue, Now a user can specify certain modules to be mocked. This would really come in handy with projects depending on packages with third party C extensions such as numpy, scipy, etc.

{% if mock_modules %}
mock_modules = [name.strip() for name in '{{ mock_modules }}'.split(',')]
sys.modules.update((mod_name, mock.Mock()) for mod_name in mock_modules)
{% endif %}

Apart from this, if we detect a setup.py in the repository or a requirements.txt, we automatically try to install from it to meet dependencies.

# autodoc imports the module while building source files. To avoid
# ImportError, install any packages in requirements.txt of the project
# if available
if [ -f $ROOT_DIR/setup.py ]; then
  pip install $ROOT_DIR/
elif [ -f $ROOT_DIR/requirements.txt ]; then
  pip install -q -r $ROOT_DIR/requirements.txt
fi

We also crawl the repository to detect any packages and add them to sys.path. With these changes, a user can expected generated API docs without having to extend conf.py.

{% if autoapi_python == 'true' %}
for (dirpath, dirnames, filenames) in os.walk('{{ root_dir }}'):
    # Directory contains __init__.py. It should be a python package
    if '__init__.py' in filenames:
        # appending instead of inserting at front so that user
        # cannot overwrite some of our own modules.
        sys.path.append(os.path.abspath(os.path.dirname(dirpath)))
{% endif %}

The second goal is a no brainer. We would like to support as many languages as we can. With this week’s update, Java has been added to the officially supported list of languages for which Yaydoc can generate full API documentation without any manual intervention. To extract API documentation for java source files, we used a sphinx extension named javasphinx. From the official javasphinx docs,

javasphinx is a Sphinx extension that provides a Sphinx domain for documenting Java projects and a javasphinx-apidoc command line tool for automatically generating API documentation from existing Java source code and Javadoc documentation.

javasphinx-apidoc -o source/ $ROOT_DIR/$AUTOAPI_JAVA_PATH/
sphinx-apidoc -o source/ $ROOT_DIR/$AUTOAPI_PYTHON_PATH/

For the third goal, we use the tools sphinx-apidoc and javasphinx-apidoc to generate source files.

Resources

Continue ReadingDocumenting APIs with Yaydoc

Documenting Open Event API Using API-Blueprint

FOSSASIA‘s Open Event Server API documentation is done using an api-blueprint. The API Blueprint language is a format used to describe API in an API blueprint file, where a blueprint file (or a set of files) is such that describes an API using the API Blueprint language. To follow up with the blueprint, an apiary editor is used. This editor is responsible for rendering the API blueprint and printing the result in user readable API documented format. We create the API blueprint manually.

Using API Blueprint:-
We create the API blueprint by first adding the name and metadata for the API we aim to design. This step looks like this :-

FORMAT: V1
HOST: https://api.eventyay.com

# Open Event API Server

The Open Event API Server

# Group Authentication

The API uses JWT Authentication to authenticate users to the server. For authentication, you need to be a registered user. Once you have registered yourself as an user, you can send a request to get the access_token.This access_token you need to then use in Authorization header while sending a request in the following manner: `Authorization: JWT <access_token>`


API blueprint starts with the metadata, here FORMAT and HOST are defined metadata. FORMAT keyword specifies the version of API Blueprint . HOST defines the host for the API.

The heading starts with # and the first heading is regarded as the name of the API.

NOTE – Also all the heading starts with one or more # symbol. Each symbol indicates the level of the heading. One # symbol followed by heading serves as the top level i.e. one # = Top Level. Similarly for  ## = second level and so on. This is in compliance with normal markdown format.
        Following the heading section comes the description of the API. Further, headings are used to break up the description section.

Resource Groups:
—————————–
    By using group keyword at the starting of a heading , we create a group of related resources. Just like in below screenshot we have created a Group Users.

# Group Users

For using the API you need(mostly) to register as an user. Registering gives you access to all non admin API endpoints. After registration, you need to create your JWT access token to send requests to the API endpoints.


| Parameter | Description | Type | Required |
|:----------|-------------|------|----------|
| `name`  | Name of the user | string | - |
| `password` | Password of the user | string | **yes** |
| `email` | Email of the user | string | **yes** |

 

Resources:
——————
    In the Group Users we have created a resource Users Collection. The heading specifies the URI used to access the resource inside of the square brackets after the heading. We have used here parameters for the resource URI which takes us into how to add parameters to the URI. Below code shows us how to add parameters to the resource URI.

## Users Collection [/v1/users{?page%5bsize%5d,page%5bnumber%5d,sort,filter}]
+ Parameters
    + page%5bsize%5d (optional, integer, `10`) - Maximum number of resources in a single paginated response.
    + page%5bnumber%5d (optional, integer, `2`) - Page number to fetchedfor the paginated response.
    + sort (optional, string, `email`) - Sort the resources according to the given attribute in ascending order. Append '-' to sort in descending order.
    + filter(optional, string, ``) - Filter according to the flask-rest-jsonapi filtering system. Please refer: http://flask-rest-jsonapi.readthedocs.io/en/latest/filtering.html for more.

 

Actions:
————–
    An action is specified with a sub-heading inside of  a resource as the name of Action, followed by HTTP method inside the square brackets.
    Before we get on further, let us discuss what a payload is. A payload is an HTTP transaction message including its discussion and any additional assets such as entity-body validation schema.

There are two payloads inside an Action:

  1. Request: It is a payload containing one specific HTTP Request, with Headers and an optional body.
  2. Response: It is a payload containing one HTTP Response.

A payload may have an identifier-a string for a request payload or an HTTP status code for a response payload.

+ Request

    + Headers

            Accept: application/vnd.api+json

            Authorization: JWT <Auth Key>

+ Response 200 (application/vnd.api+json)


Types of HTTP methods for Actions:

  • GET – In this action, we simply send the header data like Accept and Authorization and no body. Along with it we can send some GET parameters like page[size]. There are two cases for GET: List and Detail. For example, if we consider users, a GET for List helps us retrieve information about all users in the response, while Details, helps us retrieve information about a particular user.

The API Blueprint examples implementation of both GET list and detail request and response are as follows.

### List All Users [GET]
Get a list of Users.

+ Request

    + Headers

            Accept: application/vnd.api+json

            Authorization: JWT <Auth Key>

+ Response 200 (application/vnd.api+json)

        {
            "meta": {
                "count": 2
            },
            "data": [
                {
                    "attributes": {
                        "is-admin": true,
                        "last-name": null,
                        "instagram-url": null,

 

### Get Details [GET]
Get a single user.

+ Request

    + Headers

            Accept: application/vnd.api+json

            Authorization: JWT <Auth Key>

+ Response 200 (application/vnd.api+json)

        {
            "data": {
                "attributes": {
                    "is-admin": false,
                    "last-name": "Doe",
                    "instagram-url": "http://instagram.com/instagram",

 

  • POST – In this action, apart from the header information, we also need to send a data. The data must be correct with jsonapi specifications. A POST body data for an users API would look something like this:
### Create User [POST]
Create a new user using an email, password and an optional name.

+ Request (application/vnd.api+json)

    + Headers

            Authorization: JWT <Auth Key>

    + Body

            {
              "data":
              {
                "attributes":
                {
                  "email": "example@example.com",
                  "password": "password",


A POST request with this data, would create a new entry in the table and then return in jsonapi format the particular entry that was made into the table along with the id assigned to this new entry.

  • PATCH – In this action, we change or update an already existing entry in the database. So It has a header data like all other requests and a body data which is almost similar to POST except that it also needs to mention the id of the entry that we are trying to modify.
### Update User [PATCH]
+ `id` (integer) - ID of the record to update **(required)**

Update a single user by setting the email, email and/or name.

Authorized user should be same as user in request body or must be admin.

+ Request (application/vnd.api+json)

    + Headers

            Authorization: JWT <Auth Key>

    + Body

            {
              "data": {
                "attributes": {
                  "password": "password1",
                  "avatar_url": "http://example1.com/example1.png",
                  "first-name": "Jane",
                  "last-name": "Dough",
                  "details": "example1",
                  "contact": "example1",
                  "facebook-url": "http://facebook.com/facebook1",
                  "twitter-url": "http://twitter.com/twitter1",
                  "instagram-url": "http://instagram.com/instagram1",
                  "google-plus-url": "http://plus.google.com/plus.google1",
                  "thumbnail-image-url": "http://example1.com/example1.png",
                  "small-image-url": "http://example1.com/example1.png",
                  "icon-image-url": "http://example1.com/example1.png"
                },
                "type": "user",
                "id": "2"
              }
            }

Just like in POST, after we have updated our entry, we get back as response the new updated entry in the database.

  • DELETE – In this action, we delete an entry from the database. The entry in our case is soft deleted by default. Which means that instead of deleting it from the database, we set the deleted_at column with the time of deletion. For deleting we just need to send header data and send a DELETE request to the proper endpoint. If deleted successfully, we get a response as “Object successfully deleted”.
### Delete User [DELETE]
Delete a single user.

+ Request

    + Headers

            Accept: application/vnd.api+json

            Authorization: JWT <Auth Key>

+ Response 200 (application/vnd.api+json)

        {
          "meta": {
            "message": "Object successfully deleted"
          },
          "jsonapi": {
            "version": "1.0"
          }
        }


How to check after manually entering all these? We can use the
apiary website to render it, or simply use different renderer to do it. How? Checkout for my next blog on apiary and aglio.

Learn more about api blueprint here: https://apiblueprint.org/

Continue ReadingDocumenting Open Event API Using API-Blueprint

Improving Custom PyPI Theme Support In Yaydoc

Yaydoc has been supporting custom themes from nearly it’s inception. Themes, which it could not find locally, it would automatically try to install it via pip and set up appropriate metadata about the themes in the generated conf.py.  It was one of the first major enhancement we provided as compared to when using bare sphinx to generate documentation. Since then, a large number of features have been added to ease the process of documentation generation but the core theming aspects have remained unchanged.

To use a theme, sphinx needs the exact name of the theme and the absolute path to it. To obtain these metadata, the existing implementation accessed the __file__ attribute of the imported package to get the absolute path to the __init__.py file, a necessary element of all python packages. From there we searched for a file named theme.conf, and thus the directory containing that file was our required theme.

There were a few mistakes in our earlier implementation. For starters, we assumed that the distribution name of the theme in PyPI and the package name which should be imported would be same. This is generally true but is not necessary. One such theme from PyPI is Flask-Sphinx-Themes. While you need to install it using

pip install Flask-Sphinx-Themes

yet to import it in a module one needs to

import flask_sphinx_themes

This lead to build errors when specific themes like this was used. To solve this, we used the pkg_resources package. It allows us to get various metadata about a package in an abstract way without needing to specifically handle if the package is zipped or not.

try:
    dist = pkg_resources.get_distribution('{{ html_theme }}')
    top_level = list(dist._get_metadata('top_level.txt'))[0]
    dist_path = os.path.join(dist.location, top_level)
except (pkg_resources.DistributionNotFound, IndexError):
    print("\nError with distribution {0}".format('{{ html_theme }}'))
    html_theme = 'fossasia_theme'
    html_theme_path = ['_themes']

The idea here is that instead of searching for __init__.py, we read the name of the top_level directory using the first entry of the top_level.txt, a file created by setuptools when installing the package. We build the path by joining the location attribute of the Distribution object and the name of the top_level directory. The advantage with this approach is that we don’t need to import anything and thus no longer need to know the exact package name.

With this update, Support for custom themes has been greatly increased.

Resources

Continue ReadingImproving Custom PyPI Theme Support In Yaydoc

gh-pages Publishing in Yaydoc’s Web UI

A few weeks back we rolled out the web interface for yaydoc. Web UI will enable user to generate the documentation with one click and users can download the zipped format of generated documentation. In Yaydoc, we now added the additional feature of deploying the generated documentation to the GITHUB pages. In order to push the generated documentation, we have to get the access token of the user. So I used passport Github’s strategy to get the access token of the users.

passport.use(new githubStrategy({
  clientID: process.env.CLIENTID,
  clientSecret: process.env.CLIENTSECRET,
  callbackURL: process.env.CALLBACKURL
}, function (accessToken, refreshToken, profile, cb) {
  profile.token = accessToken;
  cb(null, profile)
}))

passport.serializeUser(function(user, cb) {
  cb(null, user);
});

passport.deserializeUser(function(obj, cb) {
  cb(null, obj);
}); 

After setting the necessary environment variables, we have to pass the strategy to the express handler.

router.get("/github", function (req, res, next) {
  req.session.uniqueId = req.query.uniqueId;
  req.session.email = req.query.email
  req.session.gitURL = req.query.gitURL
  next()
}, passport.authenticate('github', {
  scope: [
    'public_repo',
    'read:org'
  ]
}))

For maintaining state, I’m keeping the necessary information in the session so, that in the callback URL we know which repository have to push.

 

router.get("/callback", passport.authenticate('github'), function (req, res, next) {
  req.session.username = req.user.username;
  req.session.token = req.user.token
  res.redirect("/deploy")
})

router.get("/deploy", function (req, res, next) {
  res.render("deploy", {
    email: req.session.email,
    gitURL: req.session.gitURL,
    uniqueId: req.session.uniqueId,
    token: crypter.encrypt(req.session.token),
    username: req.session.username
  })
})

Github will send the access token to our callback. After this I’m templating the necessary information to the jade deploy template where it’ll invoke the deploy function via sockets. Then we’ll stream all the bash output log to the website.

io.on('connection', function(socket){
  socket.on('execute', function (formData) {
    generator.executeScript(socket, formData);
  });
  socket.on('deploy', function (data) {
    ghdeploy.deployPages(socket, data);
  });
});

exports.deployPages = function (socket, data) {
  var donePercent = 0;
  var repoName = data.gitURL.split("/")[4].split(".")[0];
  var webUI = "true";
  var username = data.username
  var oauthToken = crypter.decrypt(data.encryptedToken)
  const args = [
    "-e", data.email,
    "-i", data.uniqueId,
    "-w", webUI,
    "-n", username,
    "-o", oauthToken,
    "-r", repoName
  ];
  var process = spawn("./publish_docs.sh", args);

  process.stdout.on('data', function (data) {
    console.log(data.toString());
    socket.emit('deploy-logs', {donePercent: donePercent, data: data.toString()});
    donePercent += 18;
  });

  process.stderr.on('data', function (data) {
    console.log(data.toString());
    socket.emit('err-logs', data.toString());
  });

  process.on('exit', function (code) {
    console.log('child process exited with code ' + code);
    if (code === 0) {
      socket.emit('deploy-success', {pagesURL: "https://" + data.username + ".github.io/" + repoName});
    }
  });
}

Once documentation is pushed to gh-pages, the documentation URL will get appended to the web UI.

Resources:

Continue Readinggh-pages Publishing in Yaydoc’s Web UI

Using Express to show previews in Yaydoc

In yaydoc WebUI, documentation is generated using sphnix and can be downloaded as a zip file. If the user wants to see a preview of the documentation they have to unzip the zipped file and have to check the generated website for each and every build. So, we decided to implement preview feature to show the preview of generated documentation so that user will have an idea of how the website would look. Since WebUI is made with Express, we implemented the preview feature using Express’s static function. Mostly static function is used for serving static assets but we used to serve our generated site because all the generated sites are static. All the generated documentation will have an unique id and all the unique ids are generated as per uuidv4 spec. The generated document will be saved and moved to the unique folder.

mv $BUILD_DIR/_build/html $ROOT_DIR/../${UNIQUEID}_preview && cd $_/../
var express = require("express")
var path = require("path")
var favicon = require("serve-favicon");
var logger = require("morgan");
var cookieParser = require("cookie-parser");
var bodyParser = require("body-parser");
var uuidV4 = require("uuid/v4");

var app = express();
app.use(logger("dev"));
app.use(bodyParser.json()); 
app.use(bodyParser.urlencoded({ extended: false }));
app.use(cookieParser()); app.use(express.static(path.join(__dirname, "public")));
app.use("/preview", express.static(path.join(__dirname, "temp")))

app.listen(3000)

The above snippet is the just a basic Express server. In which, we have a route /preview and with the express static handler. Pass the path of your generated website as  argument, Then your sites are served over /preview route.

Resources:

Continue ReadingUsing Express to show previews in Yaydoc

Deploying documentations generated by Yaydoc to Heroku

There are many web applications available online that generates static websites. Among these projects are two unique projects developed here at FOSSASIA. These are the Open Event WebApp Generator and Yaydoc (an automatic documentation generation and deployment project.). Since Yaydoc already supports the deployment of the generated documentations to Github pages, it was just a matter of time that the deployment to Heroku is also supported.

Heroku is an excellent cloud-based platform used as a web application deployment service. Heroku provides most of its services at free of cost to the users and is excellent to host static websites provided that a little bit of tweaking is done.

For this implementation, we use the `Platform API` provided by Heroku. Stating it’s description mentioned in the documentation,

The platform API empowers developers to automate, extend and combine Heroku with other services. You can use the platform API to programmatically create apps, provision add-ons and perform other tasks that could previously only be accomplished with Heroku toolbelt or dashboard.

In order to deploy the static websites to Heroku, we need to first prepare a bundle of source code that has been compiled and is ready for execution on the Heroku runtime. This bundle is known as a Slug.

cd temp/$EMAIL/${UNIQUE_ID}_preview
mkdir -p app
cd app

curl https://nodejs.org/dist/v6.11.0/node-v6.11.0-linux-x64.tar.gz | tar xzv > /dev/null

cp $BASE/web.js
rsync -av --progress ../ . --exclude app

cd ..
tar czfv slug.tgz ./app > /dev/null

We are using the files generated for preview to bundle them in a slug. Also, we download the NodeJS runtime files since we are deploying a static website to Heroku. Along with the static files, we require bundling a NodeJS server file (web.js) that will be used to reference the static files in the application.

After preparing the Slug, we publish the static web application to Heroku. For this, we start by creating a Heroku app using the command `heroku create <app-name>`. The app name is decided by the user when he or she fills the form in the Yaydoc Web App. Following that, we request Heroku to allocate a new slug for your app. After that, we upload the slug tar file to the platform.

# Create Heroku 
heroku create $APP_NAME

# Allocating new Slug
Arr=($(curl -u “:$API_KEY” -X \
-H ‘Content-Type:application/json’ \
-H ‘Accept: application/vnd.heroku+json;version=3’ \
-d ‘{“process_types”:{“web”:”node-v6.11.0-linux-x64/bin/node web.js”}}’ \
-n https://api.heroku.com/apps/${APP_NAME}/slugs | \
python -c “import sys,json; obj=json.load(sys.stdin);
print(obj[‘blob’][‘url’] + ‘\n’ + obj[‘id’])”))

# Upload the slug tar file
curl -X PUT \
-H “Content-Type:”\
--data-binary @slug.tgz \
“${Arr[0]}”

After uploading the slug to Heroku, we need to release the app. This is done using the following command.

curl -u “:$API_KEY” -X POST \
-H “Accept: application/vnd.heroku+json; version=3” \
-H “Content-Type: application/json” \
-d ‘{“slug”:”’${Arr[1]}’”}’ \
-n https://api.heroku.com/apps/$APP_NAME/releases

Releasing the application completes the process of deployment, making the documentation generated by Yaydoc up and running at the following URL: https://<app-name>.herokuapp.com/

 

Continue ReadingDeploying documentations generated by Yaydoc to Heroku