markdown

Read more about the article Markdown Support for Experiment Docs in PSLab Android

Markdown Support for Experiment Docs in PSLab Android

The PSLab Android App and the PSLab Desktop App come with built-in experiments which include the experiment setups as well as the experiment docs. The experiment docs for PSLab have been written in the Markdown format. So, the markdown support had to be enabled in the PSLab Android App.

There are numerous markdown file renderers for android. The most popular among them is MarkdownView (https://github.com/falnatsheh/MarkdownView) which is an  open-source service.

This blog covers how to enable the support for markdown in apps and use to generate elegant documentation.

Enabling MarkdownView

MarkdownView can be enabled by simply adding a dependency in the build.gradle file

compile 'us.feras.mdv:markdownview:1.1.0'

 

Creating the layout file

The layout file for supporting a markdown file is fairly simple. The inclusion of the above dependency simplifies the things. The view holder for markdown is created and an id is assigned to it.

<?xml version="1.0" encoding="utf-8"?>
<LinearLayout
   xmlns:android="http://schemas.android.com/apk/res/android"
   xmlns:app="http://schemas.android.com/apk/res-auto"
   android:orientation="vertical"
   android:layout_width="match_parent"
   android:layout_height="match_parent">

   <br.tiagohm.markdownview.MarkdownView
       android:layout_width="match_parent"
       app:escapeHtml="false"
       android:layout_height="match_parent"
       android:id="@+id/perform_experiment_md" />
</LinearLayout>

 

Loading the markdown file

In order to load the markdown file, a MarkdownView object is created. Since, in the PSLab Android app, markdown files which form the documentation part are a part of the experiments. So, the files are displayed in the documentation fragment of the experiments.

private String mdFile;
private MarkdownView mMarkdownView;

public static ExperimentDocFragment newInstance(String mdFile) {
   ExperimentDocFragment experimentDocFragment = new ExperimentDocFragment();
   experimentDocFragment.mdFile = mdFile;
   return experimentDocFragment;
}

 

The MarkdownView object created is assigned to markdown viewholder of the relevant layout file. Here, the layout file was named experiment_doc_md and the view holder was assigned the id perform_experiment_md. The markdown files were stored in the assets directory of the app and the files were loaded from the there. 

public View onCreateView(LayoutInflater inflater, @Nullable ViewGroup container, @Nullable Bundle savedInstanceState) {
   View view = inflater.inflate(R.layout.experiment_doc_md, container, false);
   mMarkdownView = (MarkdownView) view.findViewById(R.id.perform_experiment_md);
   mMarkdownView.loadMarkdownFromAsset("capacitance.md");
   return view;
}

 

The available methods in markdown view are

  • loadMarkdown – loads directly from the content in the string 

mMarkdownView.loadMarkdown("**MarkdownView**");

 

  • loadMarkdownFromAsset – loads markdown files located in the assets directory of the app

mMarkdownView.loadMarkdownFromAsset("markdown1.md");

 

  • loadMarkdownFromFile – loads markdown from a file stored in the app not present in the assets directory

mMarkdownView.loadMarkdownFromFile(new File());

 

  • loadMarkdownFromUrl – loads markdown from the specified URL (requires internet connection, as file is loaded from the web)

mMarkdownView.loadMarkdownFromUrl("url");

 

Important points for consideration

  • Avoid using elements of GitHub Flavoured Markdown (GFM) as it is not fully supported. It is better to stick to the traditional markdown style.
  • While adding images in the markdown files, avoid using specific dimensions as the images may not load properly in some cases due to the wide variety of screen sizes in android devices.
  • It is better to store the Markdown files to be loaded in the assets directory of the app and load it from there instead of the other methods mentioned above.

References

  1. A comprehensive markdown tutorial to learn markdown scripting https://www.markdowntutorial.com/
  2. MarkdownView repository on Github by tiagohm https://github.com/tiagohm/MarkdownView
  3. Learn more about Github Flavoured Markdown (GFM) https://guides.github.com/features/mastering-markdown/
Continue ReadingMarkdown Support for Experiment Docs in PSLab Android
Read more about the article Extending Markdown Support in Yaydoc

Extending Markdown Support in Yaydoc

Yaydoc, our automatic documentation generator, builds static websites from a set of markup documents in markdown or reStructuredText format. Yaydoc uses the sphinx documentation generator internally hence reStructuredText support comes out of the box with it. To support markdown we use multiple techniques depending on the context. Most of the markdown support is provided by recommonmark, a docutils bridge for sphinx which basically converts markdown documents into proper docutil’s abstract syntax tree which is then converted to HTML by sphinx. While It works pretty well for most of the use cases, It does fall short in some instances. They are discussed in the following paragraphs.

The first problem was inclusion of other markdown files in the starting page. This was due to the fact that markdown does not supports any include mechanism. And if we used the reStructuredText include directive, the included text was parsed as reStructuredText. This problem was solved earlier using pandoc – an excellent tool to convert between various markup formats. What we did was that we created another directive mdinclude which converts the markdown to reStructuredText before inclusion. Although this was solved a while ago, The reason I’m discussing this here is that this was the inspiration behind the solution to our recent problem.

The problem we encountered was that recommonmark follows the Commonmark spec which is an ongoing effort towards standardization of markdown which has been somewhat lacking till now. The process is currently going on so the recommonmark library doesn’t yet support the concept of extensions to support various features of different markdown flavours not in the core commonmark spec. We could have settled for only supporting the markdown features in the core spec but tables not being present in the core spec was problematic. We had to support tables as it is widely used in most of the docs present in github repositories as GFM(Github Flavoured Markdown) renders ascii tables nicely.

The solution was to use a combination of recommonmark and pandoc. recommonmark provides a eval_rst code block which can be used to embed non-section reStructuredText within markdown. I created a new MarkdownParser class which inherited the CommonMarkParser class from recommonmark. Within it, using regular expressions, I convert any text within `<!– markdown+ –>` and `<!– endmarkdown+ –>`  into reStructuredText and enclose it within eval_rst code block. The result was that tables when enclosed within those trigger html comments would be converted to reST tables and then enclosed within eval_rst block which resulted in recommonmark renderering them properly. Below is a snippet which shows how this was implemented.

import re
from recommonmark.parser import CommonMarkParser
from md2rst import md2rst


MARKDOWN_PLUS_REGEX = re.compile('<!--\s+markdown\+\s+-->(.*?)<!--\s+endmarkdown\+\s+-->', re.DOTALL)
EVAL_RST_TEMPLATE = "```eval_rst\n{content}\n```"


def preprocess_markdown(inputstring):
    def callback(match_object):
        text = match_object.group(1)
        return EVAL_RST_TEMPLATE.format(content=md2rst(text))

    return re.sub(MARKDOWN_PLUS_REGEX, callback, inputstring)


class MarkdownParser(CommonMarkParser):
    def parse(self, inputstring, document):
        content = preprocess_markdown(inputstring)
        CommonMarkParser.parse(self, content, document)

Resources

Continue ReadingExtending Markdown Support in Yaydoc
Read more about the article Markdown responses from SUSI Server

Markdown responses from SUSI Server

Most of the times SUSI sends a plain text reply. But for some replies we can set the type of the query as markdown and format the output in computer or bot typed images. In this blog I will explain how to get images with markdown instead of large texts.

This servlet is a simple HttpServlet and do not require any types of user authentication or base user roles. So, instead of extending it from AbstractAPIHandler we extend a HttpServlet.

public class MarkdownServlet extends HttpServlet {

This method is fired when we send a GET request to the server. It accepts those parameters and send it to the “process(…)” method.

One major precaution in open source is to ensure no one takes advantages out of it. In the first steps, we ensure that a user is not trying to access the server very frequently. If the server find the request frequency high, it returns a 503 error to the user.

if (post.isDoS_blackout()) {response.sendError(503, "your request frequency is too high"); return;} // DoS protection
process(request, response, post);
}

 The process function is where all the processing is done. Here the text is extracted from the URL. All the parameters are sent in GET request and the “process(…)” functions parses the query. After we check all the parameters like color, padding, uppercase, text color and get them in our local variables. 

http://api.susi.ai/vis/markdown.png?text=hello%20world%0Dhello%20universe&color_text=000000&color_background=ffffff&padding=3

Here we calculate the optimum image size. A perfect size has the format 2:1, that fits into the preview window. We should not allow that the left or right border is cut away. We also resize the image here if necessary. Different clients can request different sizes of images and we can process the optimum image size here.

int lineheight = 7;
int yoffset = 0;
int height = width / 2;
while (lineheight <= 12) {
height = linecount * lineheight + 2 * padding - 1;
if (width <= 2 * height) break;
yoffset = (width / 2 - height) / 2;
height = width / 2;
lineheight++;
}

Then we print our text to the image. This is also done using the RasterPlotter. Using all the parameters that we parsed above we create a new image and set the colors, linewidth, padding etc. Here we are making a matrix with and set all the parameters that we calculated above to our image.

RasterPlotter matrix = new RasterPlotter(width, height, drawmode, color_background);
matrix.setColor(color_text);
if (c == '\n' || (c == ' ' && column + nextspace - pos >= 80)) {
x = padding - 1;
y += lineheight;
column = 0;
hashcount = 0;
if (!isFormatted) {
matrix.setColor(color_text);
}
isBold = false;
isItalic = false;
continue;
}
}

After we have our image we print the SUSI branding. Susi branding is put at the bottom right of the image. It prints “MADE WITH HTTP://SUSI.AI” at the bottom right of the image.

PrintTool.print(matrix, matrix.getWidth() - 6, matrix.getHeight() - 6, 0, "MADE WITH HTTP://SUSI.AI", 1, false, 50);

At the end we write  the image and set the cross origin access headers. This header is very important when we are using different domains on different clients. If this is not provided, the query may give the error of “Cross Origin Access blocked”.

response.addHeader("Access-Control-Allow-Origin", "*");
RemoteAccess.writeImage(fileType, response, post, matrix);
post.finalize();
}
}

This servlet can be locally tested at:

http://localhost:4000/vis/markdown.png?text=hello%20world%0Dhello%20universe&color_text=000000&color_background=ffffff&padding=3

Or at SUSI.ai API Server http://api.susi.ai/vis/markdown.png?text=hello%20world%0Dhello%20universe&color_text=000000&color_background=ffffff&padding=

Outputs

References

Oracle ImageIO Docs: https://docs.oracle.com/javase/7/docs/api/javax/imageio/ImageIO.html

Markdown Tutorial: https://github.com/adam-p/markdown-here/wiki/Markdown-Cheatsheet

Java 2D Graphics: http://docs.oracle.com/javase/tutorial/2d/

Continue ReadingMarkdown responses from SUSI Server
Read more about the article Using API Blueprint with Yaydoc

Using API Blueprint with Yaydoc

As part of extending the capability of Yaydoc to document APIs, this week we integrated API Blueprint with Yaydoc. Now we can parse apib files and add the parsed content to the generated documentation. From the official Homepage of API Blueprint,

API Blueprint is simple and accessible to everybody involved in the API lifecycle. Its syntax is concise yet expressive. With API Blueprint you can quickly design and prototype APIs to be created or document and test already deployed mission-critical APIs. It is a documentation-oriented web API description language. The API Blueprint is essentially a set of semantic assumptions laid on top of the Markdown syntax used to describe a web API.

To Integrate API Blueprint with Yaydoc, we used an sphinx extension named sphinxcontrib-apiblueprint. This extension can directly translate text in API Blueprint format into docutils nodes. The advantage with this approach as compared to using tools like aglio is that the generated html fits in nicely with the already existent theme. Though we may in future provide ability to generate html using tools like aglio if the user prefers. Adding an extension to sphinx is very easy. In the conf.py template, we added the extension to the already enabled list of extensions.

extensions += [‘sphinxcontrib.apiblueprint’]

The above extension provides a directive apiblueprint which can be then used to include apib files. The directive is very similar to the built in include directive. The difference is just that it should be only be used to include files in API Blueprint format. You can see an example below of how to use this directive.

.. apiblueprint:: <path to apib file>

Although this is enough for projects which use the ResT markup format, This cannot be used with projects using markdown as the primary markup format, since markdown doesn’t support the concept of directives. To solve this, we used the eval_rst block provided by recommonmark in Yaydoc. It allows users to embed valid ReST within markdown and recommonmark will properly parse the embedded text as ReST. Now a user can use this to use directives within markdown. You can see an example below.

```eval_rst
.. apiblueprint:: <path to apib file>
```

In order to implement this, we used the AutoStructify class provided by recommonmark. Here’s a snippet from our conf.py template. Note that this does have far reaching effects. Now users would be able to use this to add constructs like toctree in markdown which wasn’t possible before.

from recommonmark.transform import AutoStructify

def setup(app):
    app.add_config_value('recommonmark_config', {
    'enable_eval_rst': True,
    }, True)
    app.add_transform(AutoStructify)

Let’s see all of this in action. Here’s a preview of a generated documentation with API Blueprint using Yaydoc.

Resources

Continue ReadingUsing API Blueprint with Yaydoc

Adding support for Markdown in Yaydoc

Yaydoc being based on sphinx natively supports reStructuredText. From the official docs:

reStructuredText is an easy-to-read, what-you-see-is-what-you-get plaintext markup syntax and parser system. It is useful for quickly creating simple web pages, and for standalone documents. reStructuredText is designed for extensibility for specific application domains.

Although it being superior to markdown in terms of features, Markdown is still the most heavily used markup language out there. This week we added support for markdown into Yaydoc. Now you can use Markdown to document your project and Yaydoc would create a site with no changes required from your end. To achieve this, we used recommonmark, which enables sphinx to parse CommonMark, a strongly defined, highly compatible specification of Markdown. It solved most of the problem with 3 lines of code in our customized conf.py .

from recommonmark.parser import CommonMarkParser

source_parsers = {
'.md': CommonMarkParser,
}

source_suffix = ['.rst', '.md']

With this addition, sphinx can now use recommonmark to convert markdown to html. But not everything has been solved. Here is an excerpt from a previous blogpost which explains a problem yet to be solved.

Now sphinx requires an index.rst file within docs directory  which it uses to generate the first page of the site. A very obvious way to fill it which helps us avoid unnecessary duplication is to use the include directive of reStructuredText to include the README file from the root of the repository. But the Include directive can only properly include a reStructuredText, not a markdown document. Given a markdown document, it tries to parse the markdown as  reStructuredText which leads to errors.

To solve this problem, a custom directive mdinclude was created. Directives are the primary extension mechanism of reStructuredText. Most of it’s implementation is a copy of the built in Include directive from the docutils package. Before including in the doctree, mdinclude converts the docs from markdown to reStructuredText using pypandoc. The implementation is similar to the one also discussed in a previous blogpost.

class MdInclude(rst.Directive):

required_arguments = 1
optional_arguments = 0

def run(self):
    if not self.state.document.settings.file_insertion_enabled:
        raise self.warning('"%s" directive disabled.' % self.name)
    source = self.state_machine.input_lines.source(
        self.lineno - self.state_machine.input_offset - 1)
    source_dir = os.path.dirname(os.path.abspath(source))
    path = rst.directives.path(self.arguments[0])
    path = os.path.normpath(os.path.join(source_dir, path))
    path = utils.relative_path(None, path)
    path = nodes.reprunicode(path)

    encoding = self.options.get(
        'encoding', self.state.document.settings.input_encoding)
    e_handler = self.state.document.settings.input_encoding_error_handler
    tab_width = self.options.get(
        'tab-width', self.state.document.settings.tab_width)

    try:
        self.state.document.settings.record_dependencies.add(path)
        include_file = io.FileInput(source_path=path,
                                    encoding=encoding,
                                    error_handler=e_handler)
    except UnicodeEncodeError as error:
        raise self.severe('Problems with "%s" directive path:\n'
                          'Cannot encode input file path "%s" '
                          '(wrong locale?).' %
                          (self.name, SafeString(path)))
    except IOError as error:
        raise self.severe('Problems with "%s" directive path:\n%s.' %
                          (self.name, ErrorString(error)))

    try:
        rawtext = include_file.read()
    except UnicodeError as error:
        raise self.severe('Problem with "%s" directive:\n%s' %
                          (self.name, ErrorString(error)))

    output = md2rst(rawtext)
    include_lines = statemachine.string2lines(output,
                                              tab_width, 
                                              convert_whitespace=True)
    self.state_machine.insert_input(include_lines, path)
    return []

With this, Yaydoc can now be used on projects that exclusively use markdown. There are some more hurdles which we need to cross in the following weeks. Stay tuned for more updates.

Continue ReadingAdding support for Markdown in Yaydoc