How to add a custom filter to pypandoc

In Yaydoc, we met the problem of converting Markdown file into restructuredText because sphinx needs restructured text.  

Let us say we have a yml CodeBlock in Yaydoc’s README.md, but sphinx  uses pygments for code highlighting which needs yaml instead of yml for proper documentation generation. Pandoc has an excellent feature which allows us to write our own custom logic to the AST parser.

INPUT --reader--> AST --filter--> AST --writer--> OUTPUT

Let me explain this in a few steps:

  1. Initially pandoc reads the file and then converts it into nodes.
  2. Then the nodes is sent to Pandoc AST for parsing the markdown to restructuredText.
  3. The parsed node will then go to the filter. The filter is converting the parsed node according to the logic implemented.
  4. Then the Pandoc AST performs further parsing and joins the Nodes into text and is written to the file.

One important point to remember is that, Pandoc reads the conversion from the filter output stream so don’t write print statement in the filter. If you write print statement pandoc cannot  parse the JSON. In order to do debugging you can use logging module from python standard module. Here is the sample Pypandoc filter:

#!/usr/bin/env python
from pandocfilters import CodeBlock, toJSONFilter


def yml_to_yaml(key, val, fmt, meta):
    if key == 'CodeBlock':
        [[indent, classes, keyvals], code] = val
        if len(classes) > 0:
            if classes[0] == u'yml':
                classes[0] = u'yaml'
        val = [[indent, classes, keyvals], code]
        return CodeBlock(val[0], val[1])


if __name__ == "__main__":
    toJSONFilter(yml_to_yaml)

The above snippet checks whether the node is a CodeBlock or not. If it is a CodeBlock, it changes yml to yaml and prints it as a JSON in the output stream. It is then parsed by pandoc.

Finally, all you have to do is to add your filter to the Pypandoc’s filters argument.

output = pypandoc.convert_text(text, 'rst', format='md', filters=[os.path.join('filters', 'yml_filter.py')])

Resources:
http://pandoc.org/scripting.html#json-filters