[ { "path": ".gitignore", "content": "# python stuff\n*.pyc\n__pycache__/\n/dist/\nbuild/\nvenv/\n/*.egg-info\n.ropeproject/\n\n# vim tags\ntags\n\n# test stuff\ntest/output-panzer/\ntest/output-pandoc/\n\n" }, { "path": "LICENSE.txt", "content": "Copyright (c) 2015, Mark Sprevak\nAll rights reserved.\n\nRedistribution and use in source and binary forms, with or without modification,\nare permitted provided that the following conditions are met:\n\n - Redistributions of source code must retain the above copyright notice,\n this list of conditions and the following disclaimer.\n\n - Redistributions in binary form must reproduce the above copyright notice,\n this list of conditions and the following disclaimer in the documentation\n and/or other materials provided with the distribution.\n\n - Neither the name of Mark Sprevak nor the names of its contributors may\n be used to endorse or promote products derived from this software without\n specific prior written permission.\n\nTHIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS \"AS IS\" AND\nANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED\nWARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE\nDISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR\nANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES\n(INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;\nLOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON\nANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT\n(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS\nSOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.\n" }, { "path": "MANIFEST.in", "content": "include README.rst\ninclude LICENSE.txt\n" }, { "path": "README.md", "content": "-----\n\nDevelopment has ceased on panzer. Over the years, pandoc has gained\npowerful new functionality (e.g. the `--metadata-file` option and Lua\nfilters) that means that 90% of what can be done with panzer can be done\nwith pandoc and some simple wrapper scripts. I no longer use panzer in\nmy own workflow for this reason.\n\nIf you would like to take over development of panzer, let me know.\n\n-----\n\n# panzer\n\npanzer adds *styles* to\n[pandoc](http://johnmacfarlane.net/pandoc/index.html). Styles provide a\nway to set all options for a pandoc document with one line (‘I want this\ndocument be an article/CV/notes/letter’).\n\nYou can think of styles as a level up in abstraction from a pandoc\ntemplate. Styles are combinations of templates, metadata settings,\npandoc command line options, and instructions to run filters, scripts\nand postprocessors. These settings can be customised on a per writer and\nper document basis. Styles can be combined and can bear inheritance\nrelations to each other. panzer exposes a large amount of structured\ninformation to the external processes called by styles, allowing those\nprocesses to be both more powerful and themselves controllable via\nmetadata (and hence also by styles). Styles simplify makefiles, bundling\neverything related to the look of the document in one place.\n\nYou can think of panzer as an exoskeleton that sits around pandoc and\nconfigures pandoc based on a single choice in your document.\n\nTo use a style, add a field with your style name to the yaml metadata\nblock of your document:\n\n``` yaml\nstyle: Notes\n```\n\nMultiple styles can be supplied as a list:\n\n``` yaml\nstyle:\n - Notes\n - BoldHeadings\n```\n\nStyles are defined in a yaml file\n([example](https://github.com/msprev/dot-panzer/blob/master/styles/styles.yaml)).\nThe style definition file, plus associated executables, are placed in\nthe `.panzer` directory in the user’s home folder\n([example](https://github.com/msprev/dot-panzer)).\n\nA style can also be defined inside the document’s metadata block:\n\n``` yaml\n---\nstyle: Notes\nstyledef:\n Notes:\n all:\n metadata:\n numbersections: false\n latex:\n metadata:\n numbersections: true\n fontsize: 12pt\n commandline:\n columns: \"`75`\"\n lua-filter:\n - run: macroexpand.lua\n filter:\n - run: deemph.py\n...\n```\n\nStyle settings can be overridden by adding the appropriate field outside\na style definition in the document’s metadata block:\n\n``` yaml\n---\nstyle: Notes\nnumbersections: true\nfilter:\n - run: smallcaps.py\ncommandline:\n - pdf-engine: \"`xelatex`\"\n...\n```\n\n# Installation\n\n``` bash\npip3 install git+https://github.com/msprev/panzer\n```\n\n*Requirements:*\n\n - [pandoc](http://johnmacfarlane.net/pandoc/index.html) \\> 2.0\n - [Python 3](https://www.python.org/downloads/)\n - [pip](https://pip.pypa.io/en/stable/index.html) (included in most\n Python 3 distributions)\n\n*To upgrade existing installation:*\n\n``` bash\npip3 install --upgrade git+https://github.com/msprev/panzer\n```\n\nOn Arch Linux systems, the AUR package\n[panzer-git](https://aur.archlinux.org/packages/panzer-git/) can be\nused.\n\n## Troubleshooting\n\nAn [issue](https://github.com/msprev/panzer/issues/20) has been reported\nusing pip to install on Windows. If the method above does not work, use\nthe alternative install method below.\n\n``` \n git clone https://github.com/msprev/panzer\n cd panzer\n python3 setup.py install\n```\n\n*To upgrade existing installation:*\n\n``` \n cd /path/to/panzer/directory/cloned\n git pull\n python3 setup.py install --force\n```\n\n# Use\n\nRun `panzer` on your document as you would `pandoc`. If the document\nlacks a `style` field, this is equivalent to running `pandoc`. If the\ndocument has a `style` field, panzer will invoke pandoc plus any\nassociated scripts, filters, and populate the appropriate metadata\nfields.\n\n`panzer` accepts the same command line options as `pandoc`. These\noptions are passed to the underlying instance of pandoc. pandoc command\nline options can also be set via metadata.\n\npanzer has additional command line options. These are prefixed by triple\ndashes (`---`). Run the command `panzer -h` to see them:\n\n``` \n -h, --help, ---help, ---h\n show this help message and exit\n -v, --version, ---version, ---v\n show program's version number and exit\n ---quiet only print errors and warnings\n ---strict exit on first error\n ---panzer-support PANZER_SUPPORT\n panzer user data directory\n ---pandoc PANDOC pandoc executable\n ---debug DEBUG filename to write .log and .json debug files\n```\n\nPanzer expects all input and output to be utf-8.\n\n# Style definition\n\nA style definition may consist\nof:\n\n| field | value | value type |\n| :------------ | :--------------------------------- | :---------------------------- |\n| `parent` | parent(s) of style | `MetaList` or `MetaInlines` |\n| `metadata` | default metadata fields | `MetaMap` |\n| `commandline` | pandoc command line options | `MetaMap` |\n| `template` | pandoc template | `MetaInlines` or `MetaString` |\n| `preflight` | run before input doc is processed | `MetaList` |\n| `filter` | pandoc filters | `MetaList` |\n| `lua-filter` | pandoc lua filters | `MetaList` |\n| `postprocess` | run on pandoc’s output | `MetaList` |\n| `postflight` | run after output file written | `MetaList` |\n| `cleanup` | run on exit irrespective of errors | `MetaList` |\n\nStyle definitions are hierarchically structured by *name* and *writer*.\nStyle names by convention should be MixedCase (`MyNotes`) to avoid\nconfusion with other metadata fields. Writer names are the same as those\nof the relevant pandoc writer (e.g. `latex`, `html`, `docx`, etc.) A\nspecial writer, `all`, matches every writer.\n\n - `parent` takes a list or single style. Children inherit the\n properties of their parents. Children may have multiple parents.\n\n - `metadata` contains default metadata set by the style. Any metadata\n field that can appear in a pandoc document can appear here.\n\n - `commandline` specifies pandoc’s command line options.\n\n - `template` is a pandoc\n [template](http://johnmacfarlane.net/pandoc/demo/example9/templates.html)\n for the style.\n\n - `preflight` lists executables run before the document is processed.\n These are run after panzer reads the input, but before that input is\n sent to pandoc.\n\n - `filter` lists pandoc [json\n filters](http://johnmacfarlane.net/pandoc/scripting.html). Filters\n gain two new properties from panzer. For more info, see section on\n [compatibility](#compatibility) with pandoc.\n\n - `lua-filter` lists pandoc [lua\n filters](https://pandoc.org/lua-filters.html).\n\n - `postprocessor` lists executable to pipe pandoc’s output through.\n Standard unix executables (`sed`, `tr`, etc.) are examples of\n possible use. Postprocessors are skipped if a binary writer\n (e.g. `docx`) is used.\n\n - `postflight` lists executables run after the output has been\n written. If output is stdout, postflight scripts are run after\n stdout has been flushed.\n\n - `cleanup` lists executables run before panzer exits and after\n postflight scripts. Cleanup scripts run irrespective of whether an\n error has occurred earlier.\n\nExample:\n\n``` yaml\nNotes:\n all:\n metadata:\n numbersections: false\n latex:\n metadata:\n numbersections: true\n fontsize: 12pt\n commandline:\n wrap: preserve\n filter:\n - run: deemph.py\n postflight:\n - run: latexmk.py\n```\n\nIf panzer were run on the following document with the latex writer\nselected,\n\n``` yaml\n---\ntitle: \"My document\"\nstyle: Notes\n...\n```\n\nit would run pandoc with filter `deemph.py` and command line option\n`--wrap=preserve` on the following and then execute `latexmk.py`.\n\n``` yaml\n---\ntitle: \"My document\"\nnumbersections: true\nfontsize: 12pt\n...\n```\n\n## Style overriding\n\nStyles may be defined:\n\n - ‘Globally’ in `.yaml` files in `.panzer/styles/`\n - ‘Locally’ in `.yaml` files in the current working directory\n `./styles/`)\n - ‘In document’ inside a `styledef` field in the document’s yaml\n metadata block\n\nIf no `.panzer/styles/` directory is found, panzer will look for global\nstyle definitions in `.panzer/styles.yaml` if it exists. If no\n`./styles/` directory is found in the current working directory, panzer\nwill look for local style definitions in `./styles.yaml` if it exists.\n\nOverriding among style settings is determined by the following\nrules:\n\n| \\# | overriding rule |\n| :- | :----------------------------------------------------------------- |\n| 1 | Local style definitions override global style definitions |\n| 2 | In document style definitions override local style definitions |\n| 3 | Writer-specific settings override settings for `all` |\n| 4 | In a list, later styles override earlier ones |\n| 5 | Children override parents |\n| 6 | Fields set outside a style definition override any style’s setting |\n\nFor fields that pertain to scripts/filters, overriding is *additive*;\nfor other fields, it is *non-additive*:\n\n - For `metadata`, `template`, and `commandline`, if one style\n overrides another (say, a parent and child set `numbersections` to\n different values), then inheritance is non-additive, and only one\n (the child) wins.\n\n - For `preflight`, `lua-filter`, `filter`, `postflight` and `cleanup`\n if one style overrides another, then the ‘winner’ adds its items\n after those of the ‘loser’. For example, if the parent adds to\n `postflight` an item `-run: latexmk.py`, and the child adds `- run:\n printlog.py`, then `printlog.py` will be run after `latexmk.py`\n\n - To remove an item from an additive list, add it as the value of a\n `kill` field: for example, `- kill: latexmk.py`\n\nArguments passed to panzer directly on the command line trump any style\nsettings, and cannot be overridden by any metadata setting. Filters\nspecified on the command line (via `--filter` and `--lua-filter`) are\nrun first, and cannot be removed. All lua filters are run after json\nfilters. pandoc options set via panzer’s command line invocation\noverride any set via `commandline`.\n\nMultiple input files are joined according to pandoc’s rules. Metadata\nare merged using left-biased union. This means overriding behaviour when\nmerging multiple input files is different from that of panzer, and\nalways non-additive.\n\nIf fed input from stdin, panzer buffers this to a temporary file in the\ncurrent working directory before proceeding. This is required to allow\npreflight scripts to access the data. The temporary file is removed when\npanzer exits.\n\n## The run list\n\nExecutables (scripts, filters, postprocessors) are specified by a list\n(the ‘run list’). The list determines what gets run when. Processes are\nexecuted from first to last in the run list. If an item appears as the\nvalue of a `run:` field, then it is added to the run list. If an item\nappears as the value of a `kill:` field, then any previous occurrence is\nremoved from the run list. Killing an item does not prevent it from\nbeing added later. A run list can be completely emptied by adding the\nspecial item `- killall: true`.\n\nArguments can be passed to executables by listing them as the value of\nthe `args` field of that item. The value of the `args` field is passed\nas the command line options to the external process. This value of\n`args` should be a quoted inline code span (e.g. ``\"`--options`\"``) to\nprevent the parser interpreting it as markdown. Note that json filters\nalways receive the writer name as their first argument.\n\nLua filters cannot take arguments and the contents of their `args` field\nis ignored.\n\nExample:\n\n``` yaml\n- filter:\n - run: setbaseheader.py\n args: \"`--level=2`\"\n- postprocess:\n - run: sed\n args: \"`-e 's/hello/goodbye/g'`\"\n- postflight:\n - kill: open_pdf.py\n- cleanup:\n - killall: true\n```\n\nThe filter `setbaseheader.py` receives the writer name as its first\nargument and `--level=2` as its second argument.\n\nWhen panzer is searching for a filter `foo.py`, it will look for:\n\n| \\# | look for |\n| :- | :---------------------------------------------- |\n| 1 | `./foo.py` |\n| 2 | `./filter/foo.py` |\n| 3 | `./filter/foo/foo.py` |\n| 4 | `~/.panzer/filter/foo.py` |\n| 5 | `~/.panzer/filter/foo/foo.py` |\n| 6 | `foo.py` in PATH defined by current environment |\n\nSimilar rules apply to other executables and to templates.\n\nThe typical structure for the support directory `.panzer` is:\n\n .panzer/\n cleanup/\n filter/\n lua-filter/\n postflight/\n postprocess/\n preflight/\n template/\n shared/\n styles/\n\nWithin each directory, each executable may have a named subdirectory:\n\n postflight/\n latexmk/\n latexmk.py\n\n## Pandoc command line options\n\nArbitrary pandoc command line options can be set using metadata via\n`commandline`. `commandline` can appear outside a style definition and\nin a document’s metadata block, where it overrides the settings of any\nstyle.\n\n`commandline` contains one field for each pandoc command line option.\nThe field name is the unabbreviated name of the relevant pandoc command\nline option (e.g. `standalone`).\n\n - For pandoc flags, the value should be boolean (`true`, `false`),\n e.g. `standalone: true`.\n - For pandoc key-values, the value should be a quoted inline code\n span, e.g. ``include-in-header: \"`path/to/my/header`\"``.\n - For pandoc repeated key-values, the value should be a list of inline\n code spans, e.g.\n\n\n\n``` yaml\ncommandline:\n include-in-header:\n - \"`file1.txt`\"\n - \"`file2.txt`\"\n - \"`file3.txt`\"\n```\n\nRepeated key-value options in `comandline` are added after any provided\nfrom the command line. Overriding styles append to repeated key-value\nlists of the styles that they override.\n\n`false` plays a special role. `false` means that the pandoc command line\noption with the field’s name, if set, should be unset. `false` can be\nused for both flags and key-value options (e.g. `include-in-header:\nfalse`).\n\nExample:\n\n``` yaml\ncommandline:\n standalone: true\n slide-level: \"`3`\"\n number-sections: false\n include-in-header: false\n```\n\nThis passes the following options to pandoc `--standalone\n--slide-level=3` and removes any `--number-sections` and\n`--include-in-header=...` options.\n\nThese pandoc command line options cannot be set via `commandline`:\n\n - `bash-completion`\n - `dump-args`\n - `filter`\n - `from`\n - `help`\n - `ignore-args`\n - `list-extensions`\n - `list-highlight-languages`\n - `list-highlight-styles`\n - `list-input-formats`\n - `list-output-formats`\n - `lua-filter`\n - `metadata`\n - `output`\n - `print-default-data-file`\n - `print-default-template`\n - `print-highlight-style`\n - `read`\n - `template`\n - `to`\n - `variable`\n - `version`\n - `write`\n\n# Passing messages to external processes\n\nExternal processes have just as much information as panzer does. panzer\nsends its information to external processes via a json message. This\nmessage is sent as a string over stdin to scripts (preflight,\npostflight, cleanup scripts). It is stored inside a `CodeBlock` of the\nAST for filters. Note that filters need to parse the `panzer_reserved`\nfield and deserialise the contents of its `CodeBlock` to recover the\njson message. Some relevant discussion is\n[here](https://github.com/msprev/panzer/issues/38#issuecomment-367664291).\nPostprocessors do not receive a json message (if you need it, you should\nprobably be using a filter).\n\n JSON_MESSAGE = [{'metadata': METADATA,\n 'template': TEMPLATE,\n 'style': STYLE,\n 'stylefull': STYLEFULL,\n 'styledef': STYLEDEF,\n 'runlist': RUNLIST,\n 'options': OPTIONS}]\n\n - `METADATA` is a copy of the metadata branch of the document’s AST\n (useful for scripts, not useful for filters)\n\n - `TEMPLATE` is a string with path to the current template\n\n - `STYLE` is a list of current style(s)\n\n - `STYLEFULL` is a list of current style(s) including all parents,\n grandparents, etc. in order of application\n\n - `STYLEDEF` is a copy of all style definitions employed in document\n\n - `RUNLIST` is a list of processes in the run list; it has the\n following\n structure:\n\n\n\n RUNLIST = [{'kind': 'preflight'|'filter'|'lua-filter'|'postprocess'|'postflight'|'cleanup',\n 'command': 'my command',\n 'arguments': ['argument1', 'argument2', ...],\n 'status': 'queued'|'running'|'failed'|'done'\n },\n ...\n ...\n ]\n\n - `OPTIONS` is a dictionary containing panzer’s and pandoc’s command\n line options:\n\n\n\n``` python\nOPTIONS = {\n 'panzer': {\n 'panzer_support': const.DEFAULT_SUPPORT_DIR,\n 'pandoc': 'pandoc',\n 'debug': str(),\n 'quiet': False,\n 'strict': False,\n 'stdin_temp_file': str() # tempfile used to buffer stdin\n },\n 'pandoc': {\n 'input': list(), # list of input files\n 'output': '-', # output file; '-' is stdout\n 'pdf_output': False, # if pandoc will write a .pdf\n 'read': str(), # reader\n 'write': str(), # writer\n 'options': {'r': dict(), 'w': dict()}\n }\n}\n```\n\n`options` contains the command line options with which pandoc is called.\nIt consists of two separate dictionaries. The dictionary under the `'r'`\nkey contains all pandoc options pertaining to reading the source\ndocuments to the AST. The dictionary under the `'w'` key contains all\npandoc options pertaining to writing the AST to the output document.\n\nScripts read the json message above by deserialising json input on\nstdin.\n\nFilters can read the json message by reading the metadata field,\n`panzer_reserved`, stored as a raw code block in the AST, and\ndeserialising the string `JSON_MESSAGE_STR` to recover the json:\n\n panzer_reserved:\n json_message: |\n ``` {.json}\n JSON_MESSAGE_STR\n ```\n\n# Receiving messages from external processes\n\npanzer captures stderr output from all executables. This is for pretty\nprinting of info and errors. Scripts and filters should send json\nmessages to panzer via stderr. If a message is sent to stderr that is\nnot correctly formatted, panzer will print it verbatim prefixed by a\n‘\\!’.\n\nThe json message that panzer expects is a newline-separated sequence of\nutf-8 encoded json dictionaries, each with the following structure:\n\n { 'level': LEVEL, 'message': MESSAGE }\n\n - `LEVEL` is a string that sets the error level; it can take one of\n the following values:\n \n ``` \n 'CRITICAL'\n 'ERROR'\n 'WARNING'\n 'INFO'\n 'DEBUG'\n 'NOTSET'\n ```\n\n - `MESSAGE` is a string with your message\n\n# Compatibility\n\npanzer accepts pandoc filters. panzer allows filters to behave in two\nnew ways:\n\n1. Json filters can take more than one command line argument (first\n argument still reserved for the writer).\n2. A `panzer_reserved` field is added to the AST metadata branch with\n goodies for filters to mine.\n\nFor pandoc, json filters and lua-filters are applied in the order\nspecified by respective occurances of `--filter` and `--lua-filter` on\nthe command line. This behaviour is not entirely supported in panzer.\nInstead, all json filters are applied first and in the order specified\non the command line and the style definition (command line filters are\napplied first and unkillable). Then the lua-filters are applied, also in\nthe order specified on the command line and by the style definition\n(command line filters are applied first and unkillable). The reasons for\nthe divergence with pandoc’s behaviour are complex but mainly derive\nfrom performance benefit.\n\nThe follow pandoc command line options cannot be used with panzer:\n\n - `--bash-completion`\n - `--dump-args`\n - `--ignore-args`\n - `--list-extensions`\n - `--list-highlight-languages`\n - `--list-highlight-styles`\n - `--list-input-formats`\n - `--list-output-formats`\n - `--print-default-template`, `-D`\n - `--print-default-data-file`\n - `--version`, `-v`\n - `--help`, `-h`\n\nThe following metadata fields are reserved for use by panzer:\n\n - `styledef`\n - `style`\n - `template`\n - `preflight`\n - `filter`\n - `lua-filter`\n - `postflight`\n - `postprocess`\n - `cleanup`\n - `commandline`\n - `panzer_reserved`\n - `read`\n\nThe writer name `all` is also occupied.\n\n# Known issues\n\nPull requests welcome:\n\n - Slower than I would like (calls to subprocess slow in Python)\n - Calls to subprocesses (scripts, filters, etc.) block ui\n - [Possible issue under\n Windows](https://github.com/msprev/panzer/pull/9), so far reported\n by only one user. A leading dot plus slash is required on filter\n filenames. Rather than having `- run: foo.bar`, on Windows one needs\n to have `- run: ./foo.bar`. More information on this is welcome. I\n am happy to fix compatibility problems under Windows.\n\n# FAQ\n\n1. Why do I get the error `[Errno 13] Permission denied`? Filters and\n scripts must be executable. Vanilla pandoc allows filters to be run\n without their executable permission set. panzer does not allow this.\n The solution: set the executable permission of your filter or\n script, `chmod +x myfilter_name.py` For more, see\n [here](https://github.com/msprev/panzer/issues/22).\n\n2. Does panzer expand `~` or `*` inside field of a style definition?\n panzer does not do any shell expansion/globbing inside a style\n definition. The reason is described\n [here](https://github.com/msprev/panzer/issues/23). TL;DR: expansion\n and globbing are messy and not something that panzer is in a\n position to do correctly or predictably inside a style definition.\n You need to use the full path to reference your home directory\n inside a style definition.\n\n# Similar\n\n - \n - \n - \n - \n\n# Release notes\n\n - 1.4.1 (22 February 2018):\n - improved support of lua filters thanks to feedback from\n [jzeneto](https://github.com/jzeneto)\n - 1.4 (20 February 2018):\n - support added for lua filters\n - 1.3.1 (18 December 2017):\n - updated for pandoc 2.0.5\n [\\#35](https://github.com/msprev/panzer/issues/34). Support for\n all changes to command line interface and `pptx` writer.\n - 1.3 (7 November 2017):\n - updated for pandoc 2.0\n [\\#31](https://github.com/msprev/panzer/issues/31). Please note\n that this version of panzer *breaks compatibility with versions\n of pandoc earlier than 2.0*. Please upgrade to a version of\n pandoc \\>2.0. Versions of pandoc prior to 2.0 will no longer be\n supported in future releases of panzer.\n - 1.2 (12 January 2017):\n - fixed issue introduced by breaking change in panzer 1.1\n [\\#27](https://github.com/msprev/panzer/issues/27). Added panzer\n compatibility mode for pandoc versions \\<1.18. All version of\n pandoc \\>1.12.1 should work with panzer now.\n - 1.1 (27 October 2016):\n - breaking change: support pandoc 1.18’s new api; earlier versions\n of pandoc will not work\n - 1.0 (21 July 2015):\n - new: `---strict` panzer command line option:\n [\\#10](https://github.com/msprev/panzer/issues/10)\n - new: `commandline` allows repeated options using lists:\n [\\#3](https://github.com/msprev/panzer/issues/3)\n - new: `commandline` lists behave as additive in style\n inheritance: [\\#6](https://github.com/msprev/panzer/issues/6)\n - new: support multiple yaml style definition files:\n [\\#4](https://github.com/msprev/panzer/issues/4)\n - new: support local yaml style definition files:\n [\\#4](https://github.com/msprev/panzer/issues/4)\n - new: simplify format for panzer’s json message:\n [ce2a12](https://github.com/msprev/panzer/commit/f3a6cc28b78957827cb572e254977c2344ce2a12)\n - new: reproduce pandoc’s reader depending on writer settings:\n [\\#1](https://github.com/msprev/panzer/issues/1),\n [\\#7](https://github.com/msprev/panzer/issues/7)\n - fix: refactor `commandline` implementation:\n [\\#1](https://github.com/msprev/panzer/issues/1)\n - fix: improve documentation:\n [\\#2](https://github.com/msprev/panzer/issues/2)\n - fix: unicode error in `setup.py`:\n [\\#12](https://github.com/msprev/panzer/issues/12)\n - fix: support yaml style definition files without closing empty\n line: [\\#13](https://github.com/msprev/panzer/issues/13)\n - fix: add `.gitignore` files to repository:\n [PR\\#1](https://github.com/msprev/panzer/pull/9)\n - 1.0b2 (23 May 2015):\n - new: `commandline` - set arbitrary pandoc command line options\n via metadata\n - 1.0b1 (14 May 2015):\n - initial release\n" }, { "path": "README.rst", "content": "=================\npanzer user guide\n=================\n\n:Author: Mark Sprevak\n:Date: 6 November 2018\n\n--------------\n\nDevelopment has ceased on panzer. Over the years, pandoc has gained\npowerful new functionality (e.g. the ``--metadata-file`` option and Lua\nfilters) that means that 90% of what can be done with panzer can be done\nwith pandoc and some simple wrapper scripts. I no longer use panzer in\nmy own workflow for this reason.\n\nIf you would like to take over development of panzer, let me know.\n\n--------------\n\npanzer\n======\n\npanzer adds *styles* to\n`pandoc `__. Styles provide\na way to set all options for a pandoc document with one line (‘I want\nthis document be an article/CV/notes/letter’).\n\nYou can think of styles as a level up in abstraction from a pandoc\ntemplate. Styles are combinations of templates, metadata settings,\npandoc command line options, and instructions to run filters, scripts\nand postprocessors. These settings can be customised on a per writer and\nper document basis. Styles can be combined and can bear inheritance\nrelations to each other. panzer exposes a large amount of structured\ninformation to the external processes called by styles, allowing those\nprocesses to be both more powerful and themselves controllable via\nmetadata (and hence also by styles). Styles simplify makefiles, bundling\neverything related to the look of the document in one place.\n\nYou can think of panzer as an exoskeleton that sits around pandoc and\nconfigures pandoc based on a single choice in your document.\n\nTo use a style, add a field with your style name to the yaml metadata\nblock of your document:\n\n.. code:: yaml\n\n style: Notes\n\nMultiple styles can be supplied as a list:\n\n.. code:: yaml\n\n style:\n - Notes\n - BoldHeadings\n\nStyles are defined in a yaml file\n(`example `__).\nThe style definition file, plus associated executables, are placed in\nthe ``.panzer`` directory in the user’s home folder\n(`example `__).\n\nA style can also be defined inside the document’s metadata block:\n\n.. code:: yaml\n\n ---\n style: Notes\n styledef:\n Notes:\n all:\n metadata:\n numbersections: false\n latex:\n metadata:\n numbersections: true\n fontsize: 12pt\n commandline:\n columns: \"`75`\"\n lua-filter:\n - run: macroexpand.lua\n filter:\n - run: deemph.py\n ...\n\nStyle settings can be overridden by adding the appropriate field outside\na style definition in the document’s metadata block:\n\n.. code:: yaml\n\n ---\n style: Notes\n numbersections: true\n filter:\n - run: smallcaps.py\n commandline:\n - pdf-engine: \"`xelatex`\"\n ...\n\nInstallation\n============\n\n.. code:: bash\n\n pip3 install git+https://github.com/msprev/panzer\n\n*Requirements:*\n\n- `pandoc `__ > 2.0\n- `Python 3 `__\n- `pip `__ (included in most\n Python 3 distributions)\n\n*To upgrade existing installation:*\n\n.. code:: bash\n\n pip3 install --upgrade git+https://github.com/msprev/panzer\n\nOn Arch Linux systems, the AUR package\n`panzer-git `__ can be\nused.\n\nTroubleshooting\n---------------\n\nAn `issue `__ has been\nreported using pip to install on Windows. If the method above does not\nwork, use the alternative install method below.\n\n::\n\n git clone https://github.com/msprev/panzer\n cd panzer\n python3 setup.py install\n\n*To upgrade existing installation:*\n\n::\n\n cd /path/to/panzer/directory/cloned\n git pull\n python3 setup.py install --force\n\nUse\n===\n\nRun ``panzer`` on your document as you would ``pandoc``. If the document\nlacks a ``style`` field, this is equivalent to running ``pandoc``. If\nthe document has a ``style`` field, panzer will invoke pandoc plus any\nassociated scripts, filters, and populate the appropriate metadata\nfields.\n\n``panzer`` accepts the same command line options as ``pandoc``. These\noptions are passed to the underlying instance of pandoc. pandoc command\nline options can also be set via metadata.\n\npanzer has additional command line options. These are prefixed by triple\ndashes (``---``). Run the command ``panzer -h`` to see them:\n\n::\n\n -h, --help, ---help, ---h\n show this help message and exit\n -v, --version, ---version, ---v\n show program's version number and exit\n ---quiet only print errors and warnings\n ---strict exit on first error\n ---panzer-support PANZER_SUPPORT\n panzer user data directory\n ---pandoc PANDOC pandoc executable\n ---debug DEBUG filename to write .log and .json debug files\n\nPanzer expects all input and output to be utf-8.\n\nStyle definition\n================\n\nA style definition may consist of:\n\n=============== ================================== =================================\nfield value value type\n=============== ================================== =================================\n``parent`` parent(s) of style ``MetaList`` or ``MetaInlines``\n``metadata`` default metadata fields ``MetaMap``\n``commandline`` pandoc command line options ``MetaMap``\n``template`` pandoc template ``MetaInlines`` or ``MetaString``\n``preflight`` run before input doc is processed ``MetaList``\n``filter`` pandoc filters ``MetaList``\n``lua-filter`` pandoc lua filters ``MetaList``\n``postprocess`` run on pandoc’s output ``MetaList``\n``postflight`` run after output file written ``MetaList``\n``cleanup`` run on exit irrespective of errors ``MetaList``\n=============== ================================== =================================\n\nStyle definitions are hierarchically structured by *name* and *writer*.\nStyle names by convention should be MixedCase (``MyNotes``) to avoid\nconfusion with other metadata fields. Writer names are the same as those\nof the relevant pandoc writer (e.g. ``latex``, ``html``, ``docx``, etc.)\nA special writer, ``all``, matches every writer.\n\n- ``parent`` takes a list or single style. Children inherit the\n properties of their parents. Children may have multiple parents.\n\n- ``metadata`` contains default metadata set by the style. Any metadata\n field that can appear in a pandoc document can appear here.\n\n- ``commandline`` specifies pandoc’s command line options.\n\n- ``template`` is a pandoc\n `template `__\n for the style.\n\n- ``preflight`` lists executables run before the document is processed.\n These are run after panzer reads the input, but before that input is\n sent to pandoc.\n\n- ``filter`` lists pandoc `json\n filters `__. Filters\n gain two new properties from panzer. For more info, see section on\n `compatibility <#compatibility>`__ with pandoc.\n\n- ``lua-filter`` lists pandoc `lua\n filters `__.\n\n- ``postprocessor`` lists executable to pipe pandoc’s output through.\n Standard unix executables (``sed``, ``tr``, etc.) are examples of\n possible use. Postprocessors are skipped if a binary writer\n (e.g. ``docx``) is used.\n\n- ``postflight`` lists executables run after the output has been\n written. If output is stdout, postflight scripts are run after stdout\n has been flushed.\n\n- ``cleanup`` lists executables run before panzer exits and after\n postflight scripts. Cleanup scripts run irrespective of whether an\n error has occurred earlier.\n\nExample:\n\n.. code:: yaml\n\n Notes:\n all:\n metadata:\n numbersections: false\n latex:\n metadata:\n numbersections: true\n fontsize: 12pt\n commandline:\n wrap: preserve\n filter:\n - run: deemph.py\n postflight:\n - run: latexmk.py\n\nIf panzer were run on the following document with the latex writer\nselected,\n\n.. code:: yaml\n\n ---\n title: \"My document\"\n style: Notes\n ...\n\nit would run pandoc with filter ``deemph.py`` and command line option\n``--wrap=preserve`` on the following and then execute ``latexmk.py``.\n\n.. code:: yaml\n\n ---\n title: \"My document\"\n numbersections: true\n fontsize: 12pt\n ...\n\nStyle overriding\n----------------\n\nStyles may be defined:\n\n- ‘Globally’ in ``.yaml`` files in ``.panzer/styles/``\n- ‘Locally’ in ``.yaml`` files in the current working directory\n ``./styles/``)\n- ‘In document’ inside a ``styledef`` field in the document’s yaml\n metadata block\n\nIf no ``.panzer/styles/`` directory is found, panzer will look for\nglobal style definitions in ``.panzer/styles.yaml`` if it exists. If no\n``./styles/`` directory is found in the current working directory,\npanzer will look for local style definitions in ``./styles.yaml`` if it\nexists.\n\nOverriding among style settings is determined by the following rules:\n\n= ==================================================================\n# overriding rule\n= ==================================================================\n1 Local style definitions override global style definitions\n2 In document style definitions override local style definitions\n3 Writer-specific settings override settings for ``all``\n4 In a list, later styles override earlier ones\n5 Children override parents\n6 Fields set outside a style definition override any style’s setting\n= ==================================================================\n\nFor fields that pertain to scripts/filters, overriding is *additive*;\nfor other fields, it is *non-additive*:\n\n- For ``metadata``, ``template``, and ``commandline``, if one style\n overrides another (say, a parent and child set ``numbersections`` to\n different values), then inheritance is non-additive, and only one\n (the child) wins.\n\n- For ``preflight``, ``lua-filter``, ``filter``, ``postflight`` and\n ``cleanup`` if one style overrides another, then the ‘winner’ adds\n its items after those of the ‘loser’. For example, if the parent adds\n to ``postflight`` an item ``-run: latexmk.py``, and the child adds\n ``- run: printlog.py``, then ``printlog.py`` will be run after\n ``latexmk.py``\n\n- To remove an item from an additive list, add it as the value of a\n ``kill`` field: for example, ``- kill: latexmk.py``\n\nArguments passed to panzer directly on the command line trump any style\nsettings, and cannot be overridden by any metadata setting. Filters\nspecified on the command line (via ``--filter`` and ``--lua-filter``)\nare run first, and cannot be removed. All lua filters are run after json\nfilters. pandoc options set via panzer’s command line invocation\noverride any set via ``commandline``.\n\nMultiple input files are joined according to pandoc’s rules. Metadata\nare merged using left-biased union. This means overriding behaviour when\nmerging multiple input files is different from that of panzer, and\nalways non-additive.\n\nIf fed input from stdin, panzer buffers this to a temporary file in the\ncurrent working directory before proceeding. This is required to allow\npreflight scripts to access the data. The temporary file is removed when\npanzer exits.\n\nThe run list\n------------\n\nExecutables (scripts, filters, postprocessors) are specified by a list\n(the ‘run list’). The list determines what gets run when. Processes are\nexecuted from first to last in the run list. If an item appears as the\nvalue of a ``run:`` field, then it is added to the run list. If an item\nappears as the value of a ``kill:`` field, then any previous occurrence\nis removed from the run list. Killing an item does not prevent it from\nbeing added later. A run list can be completely emptied by adding the\nspecial item ``- killall: true``.\n\nArguments can be passed to executables by listing them as the value of\nthe ``args`` field of that item. The value of the ``args`` field is\npassed as the command line options to the external process. This value\nof ``args`` should be a quoted inline code span\n(e.g. :literal:`\"`--options`\"`) to prevent the parser interpreting it as\nmarkdown. Note that json filters always receive the writer name as their\nfirst argument.\n\nLua filters cannot take arguments and the contents of their ``args``\nfield is ignored.\n\nExample:\n\n.. code:: yaml\n\n - filter:\n - run: setbaseheader.py\n args: \"`--level=2`\"\n - postprocess:\n - run: sed\n args: \"`-e 's/hello/goodbye/g'`\"\n - postflight:\n - kill: open_pdf.py\n - cleanup:\n - killall: true\n\nThe filter ``setbaseheader.py`` receives the writer name as its first\nargument and ``--level=2`` as its second argument.\n\nWhen panzer is searching for a filter ``foo.py``, it will look for:\n\n= =================================================\n# look for\n= =================================================\n1 ``./foo.py``\n2 ``./filter/foo.py``\n3 ``./filter/foo/foo.py``\n4 ``~/.panzer/filter/foo.py``\n5 ``~/.panzer/filter/foo/foo.py``\n6 ``foo.py`` in PATH defined by current environment\n= =================================================\n\nSimilar rules apply to other executables and to templates.\n\nThe typical structure for the support directory ``.panzer`` is:\n\n::\n\n .panzer/\n cleanup/\n filter/\n lua-filter/\n postflight/\n postprocess/\n preflight/\n template/\n shared/\n styles/\n\nWithin each directory, each executable may have a named subdirectory:\n\n::\n\n postflight/\n latexmk/\n latexmk.py\n\nPandoc command line options\n---------------------------\n\nArbitrary pandoc command line options can be set using metadata via\n``commandline``. ``commandline`` can appear outside a style definition\nand in a document’s metadata block, where it overrides the settings of\nany style.\n\n``commandline`` contains one field for each pandoc command line option.\nThe field name is the unabbreviated name of the relevant pandoc command\nline option (e.g. ``standalone``).\n\n- For pandoc flags, the value should be boolean (``true``, ``false``),\n e.g. \\ ``standalone: true``.\n- For pandoc key-values, the value should be a quoted inline code span,\n e.g. \\ :literal:`include-in-header: \"`path/to/my/header`\"`.\n- For pandoc repeated key-values, the value should be a list of inline\n code spans, e.g.\n\n.. code:: yaml\n\n commandline:\n include-in-header:\n - \"`file1.txt`\"\n - \"`file2.txt`\"\n - \"`file3.txt`\"\n\nRepeated key-value options in ``comandline`` are added after any\nprovided from the command line. Overriding styles append to repeated\nkey-value lists of the styles that they override.\n\n``false`` plays a special role. ``false`` means that the pandoc command\nline option with the field’s name, if set, should be unset. ``false``\ncan be used for both flags and key-value options\n(e.g. ``include-in-header: false``).\n\nExample:\n\n.. code:: yaml\n\n commandline:\n standalone: true\n slide-level: \"`3`\"\n number-sections: false\n include-in-header: false\n\nThis passes the following options to pandoc\n``--standalone --slide-level=3`` and removes any ``--number-sections``\nand ``--include-in-header=...`` options.\n\nThese pandoc command line options cannot be set via ``commandline``:\n\n- ``bash-completion``\n- ``dump-args``\n- ``filter``\n- ``from``\n- ``help``\n- ``ignore-args``\n- ``list-extensions``\n- ``list-highlight-languages``\n- ``list-highlight-styles``\n- ``list-input-formats``\n- ``list-output-formats``\n- ``lua-filter``\n- ``metadata``\n- ``output``\n- ``print-default-data-file``\n- ``print-default-template``\n- ``print-highlight-style``\n- ``read``\n- ``template``\n- ``to``\n- ``variable``\n- ``version``\n- ``write``\n\nPassing messages to external processes\n======================================\n\nExternal processes have just as much information as panzer does. panzer\nsends its information to external processes via a json message. This\nmessage is sent as a string over stdin to scripts (preflight,\npostflight, cleanup scripts). It is stored inside a ``CodeBlock`` of the\nAST for filters. Note that filters need to parse the ``panzer_reserved``\nfield and deserialise the contents of its ``CodeBlock`` to recover the\njson message. Some relevant discussion is\n`here `__.\nPostprocessors do not receive a json message (if you need it, you should\nprobably be using a filter).\n\n::\n\n JSON_MESSAGE = [{'metadata': METADATA,\n 'template': TEMPLATE,\n 'style': STYLE,\n 'stylefull': STYLEFULL,\n 'styledef': STYLEDEF,\n 'runlist': RUNLIST,\n 'options': OPTIONS}]\n\n- ``METADATA`` is a copy of the metadata branch of the document’s AST\n (useful for scripts, not useful for filters)\n\n- ``TEMPLATE`` is a string with path to the current template\n\n- ``STYLE`` is a list of current style(s)\n\n- ``STYLEFULL`` is a list of current style(s) including all parents,\n grandparents, etc. in order of application\n\n- ``STYLEDEF`` is a copy of all style definitions employed in document\n\n- ``RUNLIST`` is a list of processes in the run list; it has the\n following structure:\n\n::\n\n RUNLIST = [{'kind': 'preflight'|'filter'|'lua-filter'|'postprocess'|'postflight'|'cleanup',\n 'command': 'my command',\n 'arguments': ['argument1', 'argument2', ...],\n 'status': 'queued'|'running'|'failed'|'done'\n },\n ...\n ...\n ]\n\n- ``OPTIONS`` is a dictionary containing panzer’s and pandoc’s command\n line options:\n\n.. code:: python\n\n OPTIONS = {\n 'panzer': {\n 'panzer_support': const.DEFAULT_SUPPORT_DIR,\n 'pandoc': 'pandoc',\n 'debug': str(),\n 'quiet': False,\n 'strict': False,\n 'stdin_temp_file': str() # tempfile used to buffer stdin\n },\n 'pandoc': {\n 'input': list(), # list of input files\n 'output': '-', # output file; '-' is stdout\n 'pdf_output': False, # if pandoc will write a .pdf\n 'read': str(), # reader\n 'write': str(), # writer\n 'options': {'r': dict(), 'w': dict()}\n }\n }\n\n``options`` contains the command line options with which pandoc is\ncalled. It consists of two separate dictionaries. The dictionary under\nthe ``'r'`` key contains all pandoc options pertaining to reading the\nsource documents to the AST. The dictionary under the ``'w'`` key\ncontains all pandoc options pertaining to writing the AST to the output\ndocument.\n\nScripts read the json message above by deserialising json input on\nstdin.\n\nFilters can read the json message by reading the metadata field,\n``panzer_reserved``, stored as a raw code block in the AST, and\ndeserialising the string ``JSON_MESSAGE_STR`` to recover the json:\n\n::\n\n panzer_reserved:\n json_message: |\n ``` {.json}\n JSON_MESSAGE_STR\n ```\n\nReceiving messages from external processes\n==========================================\n\npanzer captures stderr output from all executables. This is for pretty\nprinting of info and errors. Scripts and filters should send json\nmessages to panzer via stderr. If a message is sent to stderr that is\nnot correctly formatted, panzer will print it verbatim prefixed by a\n‘!’.\n\nThe json message that panzer expects is a newline-separated sequence of\nutf-8 encoded json dictionaries, each with the following structure:\n\n::\n\n { 'level': LEVEL, 'message': MESSAGE }\n\n- ``LEVEL`` is a string that sets the error level; it can take one of\n the following values:\n\n ::\n\n 'CRITICAL'\n 'ERROR'\n 'WARNING'\n 'INFO'\n 'DEBUG'\n 'NOTSET'\n\n- ``MESSAGE`` is a string with your message\n\nCompatibility\n=============\n\npanzer accepts pandoc filters. panzer allows filters to behave in two\nnew ways:\n\n1. Json filters can take more than one command line argument (first\n argument still reserved for the writer).\n2. A ``panzer_reserved`` field is added to the AST metadata branch with\n goodies for filters to mine.\n\nFor pandoc, json filters and lua-filters are applied in the order\nspecified by respective occurances of ``--filter`` and ``--lua-filter``\non the command line. This behaviour is not entirely supported in panzer.\nInstead, all json filters are applied first and in the order specified\non the command line and the style definition (command line filters are\napplied first and unkillable). Then the lua-filters are applied, also in\nthe order specified on the command line and by the style definition\n(command line filters are applied first and unkillable). The reasons for\nthe divergence with pandoc’s behaviour are complex but mainly derive\nfrom performance benefit.\n\nThe follow pandoc command line options cannot be used with panzer:\n\n- ``--bash-completion``\n- ``--dump-args``\n- ``--ignore-args``\n- ``--list-extensions``\n- ``--list-highlight-languages``\n- ``--list-highlight-styles``\n- ``--list-input-formats``\n- ``--list-output-formats``\n- ``--print-default-template``, ``-D``\n- ``--print-default-data-file``\n- ``--version``, ``-v``\n- ``--help``, ``-h``\n\nThe following metadata fields are reserved for use by panzer:\n\n- ``styledef``\n- ``style``\n- ``template``\n- ``preflight``\n- ``filter``\n- ``lua-filter``\n- ``postflight``\n- ``postprocess``\n- ``cleanup``\n- ``commandline``\n- ``panzer_reserved``\n- ``read``\n\nThe writer name ``all`` is also occupied.\n\nKnown issues\n============\n\nPull requests welcome:\n\n- Slower than I would like (calls to subprocess slow in Python)\n- Calls to subprocesses (scripts, filters, etc.) block ui\n- `Possible issue under\n Windows `__, so far reported\n by only one user. A leading dot plus slash is required on filter\n filenames. Rather than having ``- run: foo.bar``, on Windows one\n needs to have ``- run: ./foo.bar``. More information on this is\n welcome. I am happy to fix compatibility problems under Windows.\n\nFAQ\n===\n\n1. Why do I get the error ``[Errno 13] Permission denied``? Filters and\n scripts must be executable. Vanilla pandoc allows filters to be run\n without their executable permission set. panzer does not allow this.\n The solution: set the executable permission of your filter or script,\n ``chmod +x myfilter_name.py`` For more, see\n `here `__.\n\n2. Does panzer expand ``~`` or ``*`` inside field of a style definition?\n panzer does not do any shell expansion/globbing inside a style\n definition. The reason is described\n `here `__. TL;DR:\n expansion and globbing are messy and not something that panzer is in\n a position to do correctly or predictably inside a style definition.\n You need to use the full path to reference your home directory inside\n a style definition.\n\nSimilar\n=======\n\n- https://github.com/mb21/panrun\n- https://github.com/htdebeer/pandocomatic\n- https://github.com/balachia/panopy\n- https://github.com/phyllisstein/pandown\n\nRelease notes\n=============\n\n- 1.4.1 (22 February 2018):\n\n - improved support of lua filters thanks to feedback from\n `jzeneto `__\n\n- 1.4 (20 February 2018):\n\n - support added for lua filters\n\n- 1.3.1 (18 December 2017):\n\n - updated for pandoc 2.0.5\n `#35 `__. Support for\n all changes to command line interface and ``pptx`` writer.\n\n- 1.3 (7 November 2017):\n\n - updated for pandoc 2.0\n `#31 `__. Please note\n that this version of panzer *breaks compatibility with versions of\n pandoc earlier than 2.0*. Please upgrade to a version of pandoc\n >2.0. Versions of pandoc prior to 2.0 will no longer be supported\n in future releases of panzer.\n\n- 1.2 (12 January 2017):\n\n - fixed issue introduced by breaking change in panzer 1.1\n `#27 `__. Added panzer\n compatibility mode for pandoc versions <1.18. All version of\n pandoc >1.12.1 should work with panzer now.\n\n- 1.1 (27 October 2016):\n\n - breaking change: support pandoc 1.18’s new api; earlier versions\n of pandoc will not work\n\n- 1.0 (21 July 2015):\n\n - new: ``---strict`` panzer command line option:\n `#10 `__\n - new: ``commandline`` allows repeated options using lists:\n `#3 `__\n - new: ``commandline`` lists behave as additive in style\n inheritance: `#6 `__\n - new: support multiple yaml style definition files:\n `#4 `__\n - new: support local yaml style definition files:\n `#4 `__\n - new: simplify format for panzer’s json message:\n `ce2a12 `__\n - new: reproduce pandoc’s reader depending on writer settings:\n `#1 `__,\n `#7 `__\n - fix: refactor ``commandline`` implementation:\n `#1 `__\n - fix: improve documentation:\n `#2 `__\n - fix: unicode error in ``setup.py``:\n `#12 `__\n - fix: support yaml style definition files without closing empty\n line: `#13 `__\n - fix: add ``.gitignore`` files to repository:\n `PR#1 `__\n\n- 1.0b2 (23 May 2015):\n\n - new: ``commandline`` - set arbitrary pandoc command line options\n via metadata\n\n- 1.0b1 (14 May 2015):\n\n - initial release\n" }, { "path": "doc/.gitignore", "content": ".tmp/*\n*.pdf\n*.html\n\n" }, { "path": "doc/README.md", "content": "---\ntitle: \"panzer user guide\"\nauthor: Mark Sprevak\ndate: 6 November 2018\n...\n\n---\n\nDevelopment has ceased on panzer. Over the years, pandoc has gained powerful new functionality (e.g. the `--metadata-file` option and Lua filters) that means that 90% of what can be done with panzer can be done with pandoc and some simple wrapper scripts. I no longer use panzer in my own workflow for this reason.\n\nIf you would like to take over development of panzer, let me know.\n\n---\n\n# panzer\n\npanzer adds *styles* to [pandoc][].\n Styles provide a way to set all options for a pandoc document with one line ('I want this document be an article/CV/notes/letter').\n\nYou can think of styles as a level up in abstraction from a pandoc template.\n Styles are combinations of templates, metadata settings, pandoc command line options, and instructions to run filters, scripts and postprocessors.\n These settings can be customised on a per writer and per document basis.\n Styles can be combined and can bear inheritance relations to each other.\n panzer exposes a large amount of structured information to the external processes called by styles, allowing those processes to be both more powerful and themselves controllable via metadata (and hence also by styles).\n Styles simplify makefiles, bundling everything related to the look of the document in one place.\n\nYou can think of panzer as an exoskeleton that sits around pandoc and configures pandoc based on a single choice in your document.\n\nTo use a style, add a field with your style name to the yaml metadata block of your document:\n\n``` {.yaml}\nstyle: Notes\n```\n\nMultiple styles can be supplied as a list:\n\n``` {.yaml}\nstyle:\n - Notes\n - BoldHeadings\n```\n\nStyles are defined in a yaml file ([example][example-yaml]).\n The style definition file, plus associated executables, are placed in the `.panzer` directory in the user's home folder ([example][example-dot-panzer]).\n\nA style can also be defined inside the document's metadata block:\n\n``` {.yaml}\n---\nstyle: Notes\nstyledef:\n Notes:\n all:\n metadata:\n numbersections: false\n latex:\n metadata:\n numbersections: true\n fontsize: 12pt\n commandline:\n columns: \"`75`\"\n lua-filter:\n - run: macroexpand.lua\n filter:\n - run: deemph.py\n...\n```\n\nStyle settings can be overridden by adding the appropriate field outside a style definition in the document's metadata block:\n\n``` {.yaml}\n---\nstyle: Notes\nnumbersections: true\nfilter:\n - run: smallcaps.py\ncommandline:\n - pdf-engine: \"`xelatex`\"\n...\n```\n\n# Installation\n\n``` {.bash}\npip3 install git+https://github.com/msprev/panzer\n```\n\n*Requirements:*\n\n* [pandoc][] > 2.0\n* [Python 3][]\n* [pip][] (included in most Python 3 distributions)\n\n*To upgrade existing installation:*\n\n``` {.bash}\npip3 install --upgrade git+https://github.com/msprev/panzer\n```\n\nOn Arch Linux systems, the AUR package\n[panzer-git](https://aur.archlinux.org/packages/panzer-git/) can be used.\n\n## Troubleshooting\n\nAn [issue][pip-issue] has been reported using pip to install on Windows.\n If the method above does not work, use the alternative install method below.\n\n git clone https://github.com/msprev/panzer\n cd panzer\n python3 setup.py install\n\n*To upgrade existing installation:*\n\n cd /path/to/panzer/directory/cloned\n git pull\n python3 setup.py install --force\n\n# Use\n\nRun `panzer` on your document as you would `pandoc`.\n If the document lacks a `style` field, this is equivalent to running `pandoc`.\n If the document has a `style` field, panzer will invoke pandoc plus any associated scripts, filters, and populate the appropriate metadata fields.\n\n`panzer` accepts the same command line options as `pandoc`.\n These options are passed to the underlying instance of pandoc.\n pandoc command line options can also be set via metadata.\n\npanzer has additional command line options.\n These are prefixed by triple dashes (`---`).\n Run the command `panzer -h` to see them:\n\n```\n -h, --help, ---help, ---h\n show this help message and exit\n -v, --version, ---version, ---v\n show program's version number and exit\n ---quiet only print errors and warnings\n ---strict exit on first error\n ---panzer-support PANZER_SUPPORT\n panzer user data directory\n ---pandoc PANDOC pandoc executable\n ---debug DEBUG filename to write .log and .json debug files\n```\n\nPanzer expects all input and output to be utf-8.\n\n# Style definition\n\nA style definition may consist of:\n\n field value value type\n --------------- ------------------------------------ -----------------------------\n `parent` parent(s) of style `MetaList` or `MetaInlines`\n `metadata` default metadata fields `MetaMap`\n `commandline` pandoc command line options `MetaMap`\n `template` pandoc template `MetaInlines` or `MetaString`\n `preflight` run before input doc is processed `MetaList`\n `filter` pandoc filters `MetaList`\n `lua-filter` pandoc lua filters `MetaList`\n `postprocess` run on pandoc's output `MetaList`\n `postflight` run after output file written `MetaList`\n `cleanup` run on exit irrespective of errors `MetaList`\n\n\nStyle definitions are hierarchically structured by *name* and *writer*.\n Style names by convention should be MixedCase (`MyNotes`) to avoid confusion with other metadata fields.\n Writer names are the same as those of the relevant pandoc writer (e.g. `latex`, `html`, `docx`, etc.)\n A special writer, `all`, matches every writer.\n\n- `parent` takes a list or single style.\n Children inherit the properties of their parents.\n Children may have multiple parents.\n\n- `metadata` contains default metadata set by the style.\n Any metadata field that can appear in a pandoc document can appear here.\n\n- `commandline` specifies pandoc's command line options.\n\n- `template` is a pandoc [template][] for the style.\n\n- `preflight` lists executables run before the document is processed.\n These are run after panzer reads the input, but before that input is sent to pandoc.\n\n- `filter` lists pandoc [json filters][].\n Filters gain two new properties from panzer.\n For more info, see section on [compatibility](#compatibility) with pandoc.\n\n- `lua-filter` lists pandoc [lua filters](https://pandoc.org/lua-filters.html).\n\n- `postprocessor` lists executable to pipe pandoc's output through.\n Standard unix executables (`sed`, `tr`, etc.) are examples of possible use.\n Postprocessors are skipped if a binary writer (e.g. `docx`) is used.\n\n- `postflight` lists executables run after the output has been written.\n If output is stdout, postflight scripts are run after stdout has been flushed.\n\n- `cleanup` lists executables run before panzer exits and after postflight scripts.\n Cleanup scripts run irrespective of whether an error has occurred earlier.\n\nExample:\n\n``` {.yaml}\nNotes:\n all:\n metadata:\n numbersections: false\n latex:\n metadata:\n numbersections: true\n fontsize: 12pt\n commandline:\n wrap: preserve\n filter:\n - run: deemph.py\n postflight:\n - run: latexmk.py\n```\n\nIf panzer were run on the following document with the latex writer selected,\n\n``` {.yaml}\n---\ntitle: \"My document\"\nstyle: Notes\n...\n```\n\nit would run pandoc with filter `deemph.py` and command line option `--wrap=preserve` on the following and then execute `latexmk.py`.\n\n``` {.yaml}\n---\ntitle: \"My document\"\nnumbersections: true\nfontsize: 12pt\n...\n```\n\n## Style overriding\n\nStyles may be defined:\n\n- 'Globally' in `.yaml` files in `.panzer/styles/`\n- 'Locally' in `.yaml` files in the current working directory `./styles/`)\n- 'In document' inside a `styledef` field in the document's yaml metadata block\n\nIf no `.panzer/styles/` directory is found, panzer will look for global style definitions in `.panzer/styles.yaml` if it exists.\nIf no `./styles/` directory is found in the current working directory, panzer will look for local style definitions in `./styles.yaml` if it exists.\n\nOverriding among style settings is determined by the following rules:\n\n \\# overriding rule\n ---- -------------------------------------------------------------------------------\n 1 Local style definitions override global style definitions\n 2 In document style definitions override local style definitions\n 3 Writer-specific settings override settings for `all`\n 4 In a list, later styles override earlier ones\n 5 Children override parents\n 6 Fields set outside a style definition override any style's setting\n\n\nFor fields that pertain to scripts/filters, overriding is *additive*; for other fields, it is *non-additive*:\n\n- For `metadata`, `template`, and `commandline`, if one style overrides another (say, a parent and child set `numbersections` to different values),\n then inheritance is non-additive, and only one (the child) wins.\n\n- For `preflight`, `lua-filter`, `filter`, `postflight` and `cleanup` if one style overrides another, then the 'winner' adds its items after those of the 'loser'.\n For example, if the parent adds to `postflight` an item `-run: latexmk.py`, and the child adds `- run: printlog.py`,\n then `printlog.py` will be run after `latexmk.py`\n\n- To remove an item from an additive list, add it as the value of a `kill` field: for example, `- kill: latexmk.py`\n\nArguments passed to panzer directly on the command line trump any style settings, and cannot be overridden by any metadata setting.\n Filters specified on the command line (via `--filter` and `--lua-filter`) are run first, and cannot be removed.\n All lua filters are run after json filters.\n pandoc options set via panzer's command line invocation override any set via `commandline`.\n\nMultiple input files are joined according to pandoc's rules.\n Metadata are merged using left-biased union.\n This means overriding behaviour when merging multiple input files is different from that of panzer, and always non-additive.\n\nIf fed input from stdin, panzer buffers this to a temporary file in the current working directory before proceeding.\n This is required to allow preflight scripts to access the data.\n The temporary file is removed when panzer exits.\n\n## The run list\n\nExecutables (scripts, filters, postprocessors) are specified by a list (the 'run list').\n The list determines what gets run when.\n Processes are executed from first to last in the run list.\n If an item appears as the value of a `run:` field, then it is added to the run list.\n If an item appears as the value of a `kill:` field, then any previous occurrence is removed from the run list.\n Killing an item does not prevent it from being added later.\n A run list can be completely emptied by adding the special item `- killall: true`.\n\nArguments can be passed to executables by listing them as the value of the `args` field of that item.\n The value of the `args` field is passed as the command line options to the external process.\n This value of `args` should be a quoted inline code span (e.g. ``\"`--options`\"``) to prevent the parser interpreting it as markdown.\n Note that json filters always receive the writer name as their first argument.\n\nLua filters cannot take arguments and the contents of their `args` field is ignored.\n\nExample:\n\n``` {.yaml}\n- filter:\n - run: setbaseheader.py\n args: \"`--level=2`\"\n- postprocess:\n - run: sed\n args: \"`-e 's/hello/goodbye/g'`\"\n- postflight:\n - kill: open_pdf.py\n- cleanup:\n - killall: true\n```\n\nThe filter `setbaseheader.py` receives the writer name as its first argument and `--level=2` as its second argument.\n\nWhen panzer is searching for a filter `foo.py`, it will look for:\n\n # look for\n --- -------------------------------------------------\n 1 `./foo.py`\n 2 `./filter/foo.py`\n 3 `./filter/foo/foo.py`\n 4 `~/.panzer/filter/foo.py`\n 5 `~/.panzer/filter/foo/foo.py`\n 6 `foo.py` in PATH defined by current environment\n\nSimilar rules apply to other executables and to templates.\n\nThe typical structure for the support directory `.panzer` is:\n\n .panzer/\n cleanup/\n filter/\n lua-filter/\n postflight/\n postprocess/\n preflight/\n template/\n shared/\n styles/\n\nWithin each directory, each executable may have a named subdirectory:\n\n postflight/\n latexmk/\n latexmk.py\n\n## Pandoc command line options\n\nArbitrary pandoc command line options can be set using metadata via `commandline`.\n `commandline` can appear outside a style definition and in a document's metadata block,\n where it overrides the settings of any style.\n\n`commandline` contains one field for each pandoc command line option.\n The field name is the unabbreviated name of the relevant pandoc command line option (e.g. `standalone`).\n\n- For pandoc flags, the value should be boolean (`true`, `false`), e.g. `standalone: true`.\n- For pandoc key-values, the value should be a quoted inline code span, e.g. ``include-in-header: \"`path/to/my/header`\"``.\n- For pandoc repeated key-values, the value should be a list of inline code spans, e.g.\n\n``` {.yaml}\ncommandline:\n include-in-header:\n - \"`file1.txt`\"\n - \"`file2.txt`\"\n - \"`file3.txt`\"\n```\n\nRepeated key-value options in `comandline` are added after any provided from the command line.\n Overriding styles append to repeated key-value lists of the styles that they override.\n\n`false` plays a special role.\n `false` means that the pandoc command line option with the field's name, if set, should be unset.\n `false` can be used for both flags and key-value options (e.g. `include-in-header: false`).\n\nExample:\n\n``` {.yaml}\ncommandline:\n standalone: true\n slide-level: \"`3`\"\n number-sections: false\n include-in-header: false\n```\n\nThis passes the following options to pandoc `--standalone --slide-level=3` and removes any `--number-sections` and `--include-in-header=...` options.\n\nThese pandoc command line options cannot be set via `commandline`:\n\n- `bash-completion`\n- `dump-args`\n- `filter`\n- `from`\n- `help`\n- `ignore-args`\n- `list-extensions`\n- `list-highlight-languages`\n- `list-highlight-styles`\n- `list-input-formats`\n- `list-output-formats`\n- `lua-filter`\n- `metadata`\n- `output`\n- `print-default-data-file`\n- `print-default-template`\n- `print-highlight-style`\n- `read`\n- `template`\n- `to`\n- `variable`\n- `version`\n- `write`\n\n# Passing messages to external processes\n\nExternal processes have just as much information as panzer does.\n panzer sends its information to external processes via a json message.\n This message is sent as a string over stdin to scripts (preflight, postflight, cleanup scripts).\n It is stored inside a `CodeBlock` of the AST for filters.\n Note that filters need to parse the `panzer_reserved` field and deserialise the contents of its `CodeBlock` to recover the json message.\n Some relevant discussion is [here](https://github.com/msprev/panzer/issues/38#issuecomment-367664291).\n Postprocessors do not receive a json message (if you need it, you should probably be using a filter).\n\n```\nJSON_MESSAGE = [{'metadata': METADATA,\n 'template': TEMPLATE,\n 'style': STYLE,\n 'stylefull': STYLEFULL,\n 'styledef': STYLEDEF,\n 'runlist': RUNLIST,\n 'options': OPTIONS}]\n```\n\n- `METADATA` is a copy of the metadata branch of the document's AST (useful for scripts, not useful for filters)\n\n- `TEMPLATE` is a string with path to the current template\n\n- `STYLE` is a list of current style(s)\n\n- `STYLEFULL` is a list of current style(s) including all parents, grandparents, etc. in order of application\n\n- `STYLEDEF` is a copy of all style definitions employed in document\n\n- `RUNLIST` is a list of processes in the run list; it has the following structure:\n\n```\nRUNLIST = [{'kind': 'preflight'|'filter'|'lua-filter'|'postprocess'|'postflight'|'cleanup',\n 'command': 'my command',\n 'arguments': ['argument1', 'argument2', ...],\n 'status': 'queued'|'running'|'failed'|'done'\n },\n ...\n ...\n ]\n```\n\n- `OPTIONS` is a dictionary containing panzer's and pandoc's command line options:\n\n``` {.python}\nOPTIONS = {\n 'panzer': {\n 'panzer_support': const.DEFAULT_SUPPORT_DIR,\n 'pandoc': 'pandoc',\n 'debug': str(),\n 'quiet': False,\n 'strict': False,\n 'stdin_temp_file': str() # tempfile used to buffer stdin\n },\n 'pandoc': {\n 'input': list(), # list of input files\n 'output': '-', # output file; '-' is stdout\n 'pdf_output': False, # if pandoc will write a .pdf\n 'read': str(), # reader\n 'write': str(), # writer\n 'options': {'r': dict(), 'w': dict()}\n }\n}\n```\n\n`options` contains the command line options with which pandoc is called.\nIt consists of two separate dictionaries.\nThe dictionary under the `'r'` key contains all pandoc options pertaining to reading the source documents to the AST.\nThe dictionary under the `'w'` key contains all pandoc options pertaining to writing the AST to the output document.\n\nScripts read the json message above by deserialising json input on stdin.\n\nFilters can read the json message by reading the metadata field, `panzer_reserved`, stored as a raw code block in the AST, and deserialising the string `JSON_MESSAGE_STR` to recover the json:\n\n panzer_reserved:\n json_message: |\n ``` {.json}\n JSON_MESSAGE_STR\n ```\n\n# Receiving messages from external processes\n\npanzer captures stderr output from all executables.\n This is for pretty printing of info and errors.\n Scripts and filters should send json messages to panzer via stderr.\n If a message is sent to stderr that is not correctly formatted, panzer will print it verbatim prefixed by a '!'.\n\nThe json message that panzer expects is a newline-separated sequence of utf-8 encoded json dictionaries, each with the following structure:\n\n { 'level': LEVEL, 'message': MESSAGE }\n\n- `LEVEL` is a string that sets the error level; it can take one of the following values:\n\n 'CRITICAL'\n 'ERROR'\n 'WARNING'\n 'INFO'\n 'DEBUG'\n 'NOTSET'\n\n- `MESSAGE` is a string with your message\n\n\n# Compatibility\n\npanzer accepts pandoc filters.\n panzer allows filters to behave in two new ways:\n\n1. Json filters can take more than one command line argument (first argument still reserved for the writer).\n2. A `panzer_reserved` field is added to the AST metadata branch with goodies for filters to mine.\n\nFor pandoc, json filters and lua-filters are applied in the order specified by respective occurances of `--filter` and `--lua-filter` on the command line.\n This behaviour is not entirely supported in panzer.\n Instead, all json filters are applied first and in the order specified on the command line and the style definition (command line filters are applied first and unkillable).\n Then the lua-filters are applied, also in the order specified on the command line and by the style definition (command line filters are applied first and unkillable).\n The reasons for the divergence with pandoc's behaviour are complex but mainly derive from performance benefit.\n\nThe follow pandoc command line options cannot be used with panzer:\n\n- `--bash-completion`\n- `--dump-args`\n- `--ignore-args`\n- `--list-extensions`\n- `--list-highlight-languages`\n- `--list-highlight-styles`\n- `--list-input-formats`\n- `--list-output-formats`\n- `--print-default-template`, `-D`\n- `--print-default-data-file`\n- `--version`, `-v`\n- `--help`, `-h`\n\nThe following metadata fields are reserved for use by panzer:\n\n* `styledef`\n* `style`\n* `template`\n* `preflight`\n* `filter`\n* `lua-filter`\n* `postflight`\n* `postprocess`\n* `cleanup`\n* `commandline`\n* `panzer_reserved`\n* `read`\n\nThe writer name `all` is also occupied.\n\n# Known issues\n\nPull requests welcome:\n\n* Slower than I would like (calls to subprocess slow in Python)\n* Calls to subprocesses (scripts, filters, etc.) block ui\n* [Possible issue under Windows](https://github.com/msprev/panzer/pull/9), so far reported by only one user.\n A leading dot plus slash is required on filter filenames.\n Rather than having `- run: foo.bar`, on Windows\n one needs to have `- run: ./foo.bar`.\n More information on this is welcome.\n I am happy to fix compatibility problems under Windows.\n\n# FAQ\n\n1. Why do I get the error `[Errno 13] Permission denied`?\n Filters and scripts must be executable.\n Vanilla pandoc allows filters to be run without their executable permission set.\n panzer does not allow this.\n The solution: set the executable permission of your filter or script, `chmod +x myfilter_name.py`\n For more, see [here](https://github.com/msprev/panzer/issues/22).\n\n2. Does panzer expand `~` or `*` inside field of a style definition?\n panzer does not do any shell expansion/globbing inside a style definition.\n The reason is described [here](https://github.com/msprev/panzer/issues/23).\n TL;DR: expansion and globbing are messy and not something that panzer is in a position to do correctly or predictably inside a style definition.\n You need to use the full path to reference your home directory inside a style definition.\n\n\n# Similar\n\n* \n* \n* \n* \n\n# Release notes\n\n- 1.4.1 (22 February 2018):\n - improved support of lua filters thanks to feedback from [jzeneto](https://github.com/jzeneto)\n- 1.4 (20 February 2018):\n - support added for lua filters\n- 1.3.1 (18 December 2017):\n - updated for pandoc 2.0.5 [#35](https://github.com/msprev/panzer/issues/34). Support for all changes to command line interface and `pptx` writer.\n- 1.3 (7 November 2017):\n - updated for pandoc 2.0 [#31](https://github.com/msprev/panzer/issues/31). Please note that this version of panzer *breaks compatibility with versions of pandoc earlier than 2.0*. Please upgrade to a version of pandoc >2.0. Versions of pandoc prior to 2.0 will no longer be supported in future releases of panzer.\n- 1.2 (12 January 2017):\n - fixed issue introduced by breaking change in panzer 1.1 [#27](https://github.com/msprev/panzer/issues/27). Added panzer compatibility mode for pandoc versions <1.18. All version of pandoc >1.12.1 should work with panzer now.\n\n- 1.1 (27 October 2016):\n - breaking change: support pandoc 1.18's new api; earlier versions of pandoc will not work\n\n- 1.0 (21 July 2015):\n - new: `---strict` panzer command line option: [#10](https://github.com/msprev/panzer/issues/10)\n - new: `commandline` allows repeated options using lists: [#3](https://github.com/msprev/panzer/issues/3)\n - new: `commandline` lists behave as additive in style inheritance: [#6](https://github.com/msprev/panzer/issues/6)\n - new: support multiple yaml style definition files: [#4](https://github.com/msprev/panzer/issues/4)\n - new: support local yaml style definition files: [#4](https://github.com/msprev/panzer/issues/4)\n - new: simplify format for panzer's json message: [ce2a12](https://github.com/msprev/panzer/commit/f3a6cc28b78957827cb572e254977c2344ce2a12)\n - new: reproduce pandoc's reader depending on writer settings: [#1](https://github.com/msprev/panzer/issues/1), [#7](https://github.com/msprev/panzer/issues/7)\n - fix: refactor `commandline` implementation: [#1](https://github.com/msprev/panzer/issues/1)\n - fix: improve documentation: [#2](https://github.com/msprev/panzer/issues/2)\n - fix: unicode error in `setup.py`: [#12](https://github.com/msprev/panzer/issues/12)\n - fix: support yaml style definition files without closing empty line: [#13](https://github.com/msprev/panzer/issues/13)\n - fix: add `.gitignore` files to repository: [PR#1](https://github.com/msprev/panzer/pull/9)\n\n- 1.0b2 (23 May 2015):\n - new: `commandline` - set arbitrary pandoc command line options via metadata\n\n- 1.0b1 (14 May 2015):\n - initial release\n\n [pandoc]: http://johnmacfarlane.net/pandoc/index.html\n [panzer]: https://github.com/msprev\n [python 3]: https://www.python.org/downloads/\n [json filters]: http://johnmacfarlane.net/pandoc/scripting.html\n [template]: http://johnmacfarlane.net/pandoc/demo/example9/templates.html\n [example-yaml]: https://github.com/msprev/dot-panzer/blob/master/styles/styles.yaml\n [example-dot-panzer]: https://github.com/msprev/dot-panzer\n [setuptools for Python3]: http://stackoverflow.com/questions/14426491/python-3-importerror-no-module-named-setuptools\n [pip]: https://pip.pypa.io/en/stable/index.html\n [pip-issue]: https://github.com/msprev/panzer/issues/20\n\n" }, { "path": "doc/makefile", "content": "SOURCE = README.md\n\nPANDOC = /usr/local/bin/pandoc\nPANZER = /usr/local/bin/panzer\nTEMPLATES = /Users/msprevak/Dropbox/msprevak/dot-panzer/template\nOUT_BASENAME := $(basename $(SOURCE))\n\n.PHONY: clean\n\n########################################################################\n# Make the documentation files for release #\n########################################################################\n\ndoc:\t$(SOURCE)\n\t$(PANDOC) $(SOURCE) -o ../$(OUT_BASENAME).md -t gfm --standalone; \\\n\t$(PANDOC) $(SOURCE) -o ../$(OUT_BASENAME).rst -t rst --standalone\n\n########################################################################\n# Standard makefile for markdown file #\n########################################################################\n\nhtml:\t$(SOURCE)\n\t$(PANZER) $(SOURCE) -o $(OUT_BASENAME).html\n\npdf:\t$(SOURCE)\n\t$(PANZER) $(SOURCE) -o $(OUT_BASENAME).tex\n\ndocx:\t$(SOURCE)\n\t$(PANZER) $(SOURCE) -o $(OUT_BASENAME).docx\n\npandoc-html:\t$(SOURCE)\n\t$(PANDOC) \\\n\t\t$(SOURCE) \\\n\t\t-o .tmp/_$(OUT_BASENAME)_.html \\\n\t\t--template $(TEMPLATES)/article.html \\\n\t\t-H $(TEMPLATES)/Momento-pandoc.css \\\n\t\t--smart \\\n\t\t--standalone\n\nclean:\n\t- rm -f .tmp/* \\\n $(OUT_BASENAME).pdf $(OUT_BASENAME).docx $(OUT_BASENAME).html \\\n\t\t../$(OUT_BASENAME).md \\\n\t\t../$(OUT_BASENAME).rst\n" }, { "path": "panzer/__init__.py", "content": "\"\"\" check that python3 is being used \"\"\"\nimport sys\n\nif sys.version_info[0] != 3:\n print(\"panzer cannot run --- it requires Python 3\")\n sys.exit(1)\n" }, { "path": "panzer/cli.py", "content": "\"\"\" command line options for panzer \"\"\"\nimport argparse\nimport os\nimport shutil\nimport sys\nimport tempfile\nfrom . import const\nfrom . import version\n\nPANZER_DESCRIPTION = '''\nPanzer-specific arguments are prefixed by triple dashes ('---').\nOther arguments are passed to pandoc.\n\n panzer default user data directory: \"%s\"\n pandoc default executable: \"%s\"\n''' % (const.DEFAULT_SUPPORT_DIR, shutil.which('pandoc'))\n\nPANZER_EPILOG = '''\nCopyright (C) 2017 Mark Sprevak\nWeb: http://sites.google.com/site/msprevak\nThis is free software; see the source for copying conditions. There is no\nwarranty, not even for merchantability or fitness for a particular purpose.\n'''\n\ndef parse_cli_options(options):\n \"\"\" parse command line options \"\"\"\n #\n # disable pylint warnings:\n # + Too many local variables (too-many-locals)\n # + Too many branches (too-many-branches)\n # pylint: disable=R0912\n # pylint: disable=R0914\n #\n # 1. Parse options specific to panzer\n panzer_known, unknown = panzer_parse()\n # 2. Update options with panzer-specific values\n for field in panzer_known:\n val = panzer_known[field]\n if val:\n options['panzer'][field] = val\n # 3. Parse options specific to pandoc\n pandoc_known, unknown = pandoc_parse(unknown)\n # 2. Update options with pandoc-specific values\n for field in pandoc_known:\n val = pandoc_known[field]\n if val:\n options['pandoc'][field] = val\n # 3. Check for pandoc output being pdf\n if os.path.splitext(options['pandoc']['output'])[1].lower() == '.pdf':\n options['pandoc']['pdf_output'] = True\n # 4. Detect pandoc's writer\n # - first case: writer explicitly specified by cli option\n if options['pandoc']['write']:\n pass\n # - second case: html default writer for stdout\n elif options['pandoc']['output'] == '-':\n options['pandoc']['write'] = 'html'\n # - third case: writer set via output filename extension\n else:\n ext = os.path.splitext(options['pandoc']['output'])[1].lower()\n implicit_writer = const.PANDOC_WRITER_MAPPING.get(ext)\n if implicit_writer is not None:\n options['pandoc']['write'] = implicit_writer\n else:\n # - html is default writer for unrecognised extensions\n options['pandoc']['write'] = 'html'\n # 5. Input from stdin\n # - if one of the inputs is stdin then read from stdin now into\n # - temp file, then replace '-'s in input filelist with reference to file\n if '-' in options['pandoc']['input']:\n # Read from stdin now into temp file in cwd\n stdin_bytes = sys.stdin.buffer.read()\n with tempfile.NamedTemporaryFile(prefix='__panzer-',\n suffix='__',\n dir=os.getcwd(),\n delete=False) as temp_file:\n temp_filename = os.path.join(os.getcwd(), temp_file.name)\n options['panzer']['stdin_temp_file'] = temp_filename\n temp_file.write(stdin_bytes)\n temp_file.flush()\n # Replace all reference to stdin in pandoc cli with temp file\n for index, val in enumerate(options['pandoc']['input']):\n if val == '-':\n options['pandoc']['input'][index] = options['panzer']['stdin_temp_file']\n # 6. Remaining options for pandoc\n opt_known, unknown = pandoc_opt_parse(unknown)\n # - sort them into reader and writer phase options\n for opt in opt_known:\n # undo weird transform that argparse does to match option name\n # https://docs.python.org/dev/library/argparse.html#dest\n opt_name = str(opt).replace('_', '-')\n if opt_name not in const.PANDOC_OPT_PHASE:\n print('ERROR: '\n 'do not know reader/writer type of command line option \"--%s\"'\n '---ignoring' % opt_name)\n continue\n for phase in const.PANDOC_OPT_PHASE[opt_name]:\n options['pandoc']['options'][phase][opt_name] = opt_known[opt]\n # cli option is mutable by `commandline` metadata if\n # - not set ( == None or == False)\n # - of type list\n if opt_known[opt] == None \\\n or opt_known[opt] == False \\\n or type(opt_known[opt]) is list:\n options['pandoc']['mutable'][phase][opt_name] = True\n else:\n options['pandoc']['mutable'][phase][opt_name] = False\n\n options['pandoc'] = set_quirky_dependencies(options['pandoc'])\n # 7. print error messages for unknown options\n for opt in unknown:\n if opt in const.PANDOC_BAD_OPTS:\n print('ERROR: '\n 'pandoc command line option \"%s\" not supported by panzer'\n '---ignoring' % opt)\n else:\n print('ERROR: '\n 'do not recognize command line option \"%s\"'\n '---ignoring' % opt)\n return options\n\ndef panzer_parse():\n \"\"\" return list of arguments recognised by panzer + unknowns \"\"\"\n panzer_parser = argparse.ArgumentParser(\n description=PANZER_DESCRIPTION,\n epilog=PANZER_EPILOG,\n formatter_class=argparse.RawTextHelpFormatter,\n add_help=False)\n panzer_parser.add_argument(\"-h\", \"--help\", '---help', '---h',\n action=\"help\",\n help=\"show this help message and exit\")\n panzer_parser.add_argument('-v', '--version', '---version', '---v',\n action='version',\n version=('%(prog)s ' + version.VERSION))\n panzer_parser.add_argument(\"---quiet\",\n action='store_true',\n help='only print errors and warnings')\n panzer_parser.add_argument(\"---strict\",\n action='store_true',\n help='exit on first error')\n panzer_parser.add_argument(\"---panzer-support\",\n help='panzer user data directory')\n panzer_parser.add_argument(\"---pandoc\",\n help='pandoc executable')\n panzer_parser.add_argument(\"---debug\",\n help='filename to write .log and .json debug files')\n panzer_known_raw, unknown = panzer_parser.parse_known_args()\n panzer_known = vars(panzer_known_raw)\n return (panzer_known, unknown)\n\ndef pandoc_parse(args):\n \"\"\" return list of arguments recognised by pandoc + unknowns \"\"\"\n pandoc_parser = argparse.ArgumentParser(prog='pandoc')\n pandoc_parser.add_argument('input', nargs='*')\n pandoc_parser.add_argument(\"--read\", \"-r\", \"--from\", \"-f\")\n pandoc_parser.add_argument(\"--write\", \"-w\", \"--to\", \"-t\")\n pandoc_parser.add_argument(\"--output\", \"-o\")\n pandoc_parser.add_argument(\"--template\")\n pandoc_parser.add_argument(\"--filter\", nargs=1, action='append')\n pandoc_parser.add_argument(\"--lua-filter\", nargs=1, action='append')\n pandoc_known_raw, unknown = pandoc_parser.parse_known_args(args)\n pandoc_known = vars(pandoc_known_raw)\n return (pandoc_known, unknown)\n\ndef pandoc_opt_parse(args):\n \"\"\" return list of pandoc command line options \"\"\"\n opt_parser = argparse.ArgumentParser(prog='pandoc')\n # general options\n opt_parser.add_argument(\"--data-dir\")\n # reader options\n opt_parser.add_argument('--abbreviations')\n opt_parser.add_argument('--base-header-level')\n opt_parser.add_argument('--default-image-extension')\n opt_parser.add_argument('--extract-media')\n opt_parser.add_argument('--file-scope', action='store_true')\n opt_parser.add_argument('--indented-code-classes')\n opt_parser.add_argument('--metadata', '-M', nargs=1, action='append')\n opt_parser.add_argument('--old-dashes', action='store_true')\n opt_parser.add_argument('--preserve-tabs', '-p', action='store_true')\n opt_parser.add_argument('--tab-stop')\n opt_parser.add_argument('--track-changes')\n # writer options\n opt_parser.add_argument('--ascii', action='store_true')\n opt_parser.add_argument('--atx-headers', action='store_true')\n opt_parser.add_argument('--biblatex', action='store_true')\n opt_parser.add_argument('--bibliography', nargs=1, action='append')\n opt_parser.add_argument('--chapters', action='store_true')\n opt_parser.add_argument('--citation-abbreviations')\n opt_parser.add_argument('--columns')\n opt_parser.add_argument('--csl')\n opt_parser.add_argument('--css', '-c', nargs=1, action='append')\n opt_parser.add_argument('--dpi')\n opt_parser.add_argument('--email-obfuscation')\n opt_parser.add_argument('--eol')\n opt_parser.add_argument('--epub-chapter-level')\n opt_parser.add_argument('--epub-cover-image')\n opt_parser.add_argument('--epub-embed-font')\n opt_parser.add_argument('--epub-metadata')\n opt_parser.add_argument('--epub-subdirectory')\n opt_parser.add_argument('--gladtex', action='store_true')\n opt_parser.add_argument('--highlight-style')\n opt_parser.add_argument('--html-q-tags', action='store_true')\n opt_parser.add_argument('--id-prefix')\n opt_parser.add_argument('--include-after-body', '-A', nargs=1, action='append')\n opt_parser.add_argument('--include-before-body', '-B', nargs=1, action='append')\n opt_parser.add_argument('--include-in-header', '-H', nargs=1, action='append')\n opt_parser.add_argument('--incremental', '-i', action='store_true')\n opt_parser.add_argument('--jsmath')\n opt_parser.add_argument('--katex')\n opt_parser.add_argument('--katex-stylesheet')\n opt_parser.add_argument('--pdf-engine')\n opt_parser.add_argument('--pdf-engine-opt', nargs=1, action='append')\n opt_parser.add_argument('--latexmathml', '-m')\n opt_parser.add_argument('--listings', action='store_true')\n opt_parser.add_argument('--log')\n opt_parser.add_argument('--mathjax')\n opt_parser.add_argument('--mathml', action='store_true')\n opt_parser.add_argument('--mimetex')\n opt_parser.add_argument('--natbib', action='store_true')\n opt_parser.add_argument('--no-highlight', action='store_true')\n opt_parser.add_argument('--no-tex-ligatures', action='store_true')\n opt_parser.add_argument('--no-wrap', action='store_true')\n opt_parser.add_argument('--number-offset')\n opt_parser.add_argument('--number-sections', '-N', action='store_true')\n opt_parser.add_argument('--reference-doc')\n opt_parser.add_argument('--reference-links', action='store_true')\n opt_parser.add_argument('--reference-location')\n opt_parser.add_argument('--request-header')\n opt_parser.add_argument('--resource-path')\n opt_parser.add_argument('--section-divs', action='store_true')\n opt_parser.add_argument('--self-contained', action='store_true')\n opt_parser.add_argument('--slide-level')\n opt_parser.add_argument('--standalone', '-s', action='store_true')\n opt_parser.add_argument('--syntax-definition')\n opt_parser.add_argument('--table-of-contents', '--toc', action='store_true')\n opt_parser.add_argument('--title-prefix', '-T')\n opt_parser.add_argument('--toc-depth')\n opt_parser.add_argument('--top-level-division')\n opt_parser.add_argument('--variable', '-V', nargs=1, action='append')\n opt_parser.add_argument('--verbose', action='store_true')\n opt_parser.add_argument('--webtex')\n opt_parser.add_argument('--wrap')\n opt_known_raw, unknown = opt_parser.parse_known_args(args)\n opt_known = vars(opt_known_raw)\n return (opt_known, unknown)\n\ndef set_quirky_dependencies(pandoc):\n \"\"\" Set defaults for pandoc options that are dependent in a quirky way,\n and that panzer route via json would disrupt.\n Quirky here means that pandoc would have to know the writer to\n set the reader to the correct defaults or vice versa \"\"\"\n # --smart: reader setting\n # True when the output format is latex or context, unless --no-tex-ligatures\n # is used.\n # if (pandoc['write'] == 'latex' or \\\n # pandoc['write'] == 'beamer' or \\\n # pandoc['write'] == 'context') \\\n # and pandoc['options']['w']['no-tex-ligatures'] == False:\n # pandoc['options']['r']['smart'] = True\n ## this is commented out as apparently not needed with pandoc >2.0\n return pandoc\n\n" }, { "path": "panzer/const.py", "content": "\"\"\" constants for panzer code \"\"\"\nimport os\n\nDEBUG_TIMING = False\n\nUSE_OLD_API = False\nREQUIRE_PANDOC_ATLEAST = \"2.0\"\n\nDEFAULT_SUPPORT_DIR = os.path.join(os.path.expanduser('~'), '.panzer')\n\nENCODING = 'utf8'\n\n# keys to access type and content of metadata fields\nT = 't'\nC = 'c'\n\n# list of 'kind' of items on runlist, in order they should run\nRUNLIST_KIND = ['preflight', 'filter', 'lua-filter', 'postprocess', 'postflight', 'cleanup']\n\n# 'status' of items on runlist\nQUEUED = 'queued'\nRUNNING = 'running'\nFAILED = 'failed'\nDONE = 'done'\n\n# ast of an empty pandoc document\nEMPTY_DOCUMENT = {\"blocks\":[],\"pandoc-api-version\":[1,17,0,4],\"meta\":{}}\nEMPTY_DOCUMENT_OLDAPI = [{'unMeta': {}}, []]\n\n# writers that give binary outputs\n# these cannot be written to stdout\nBINARY_WRITERS = ['odt', 'docx', 'epub', 'epub3', 'pptx']\n\n# forbidden options for panzer command line\nPANDOC_BAD_OPTS = [\n '--bash-completion',\n '--dump-args',\n '--ignore-args',\n '--list-extensions',\n '--list-highlight-languages',\n '--list-highlight-styles',\n '--list-input-formats',\n '--list-output-formats',\n '--print-default-data-file',\n '--print-default-template',\n '--print-highlight-style',\n '-D'\n]\n\n# forbidden options for 'commandline' metadata field\nPANDOC_BAD_COMMANDLINE = [\n 'bash-completion',\n 'dump-args',\n 'filter',\n 'from',\n 'help',\n 'ignore-args',\n 'list-extensions',\n 'list-highlight-languages',\n 'list-highlight-styles',\n 'list-input-formats',\n 'list-output-formats',\n 'lua-filter',\n 'metadata',\n 'output',\n 'print-default-data-file',\n 'print-default-template',\n 'print-highlight-style',\n 'read',\n 'template',\n 'to',\n 'variable',\n 'version',\n 'write'\n]\n\n# additive command line options\nPANDOC_OPT_ADDITIVE = ['metadata',\n 'variable',\n 'bibliography',\n 'include-in-header',\n 'include-before-body',\n 'include-after-body',\n 'css',\n 'pdf-engine-opt']\n\n# pandoc's command line options, divided by reader or writer\nPANDOC_OPT_PHASE = {\n # general options\n 'data-dir': 'rw',\n 'log': 'rw',\n # reader options\n 'abbreviations': 'r',\n 'base-header-level': 'r',\n 'bibliography': 'r',\n 'citation-abbreviations': 'r',\n 'csl': 'r',\n 'default-image-extension': 'r',\n 'extract-media': 'r',\n 'file-scope': 'r',\n 'indented-code-classes': 'r',\n 'metadata': 'r',\n 'metadata-file': 'r',\n 'old-dashes': 'r',\n 'preserve-tabs': 'r',\n 'strip-empty-paragraphs': 'r',\n 'tab-stop': 'r',\n 'track-changes': 'r',\n # writer options\n 'ascii': 'w',\n 'atx-headers': 'w',\n 'biblatex': 'w',\n 'chapters': 'w',\n 'columns': 'w',\n 'css': 'w',\n 'dpi': 'w',\n 'email-obfuscation': 'w',\n 'eol': 'w',\n 'epub-chapter-level': 'w',\n 'epub-cover-image': 'w',\n 'epub-embed-font': 'w',\n 'epub-metadata': 'w',\n 'epub-subdirectory': 'w',\n 'gladtex': 'w',\n 'highlight-style': 'w',\n 'html-q-tags': 'w',\n 'id-prefix': 'w',\n 'include-after-body': 'w',\n 'include-before-body': 'w',\n 'include-in-header': 'w',\n 'incremental': 'w',\n 'jsmath': 'w',\n 'katex': 'w',\n 'katex-stylesheet': 'w',\n 'latexmathml': 'w',\n 'listings': 'w',\n 'mathjax': 'w',\n 'mathml': 'w',\n 'mimetex': 'w',\n 'natbib': 'w',\n 'no-highlight': 'w',\n 'no-tex-ligatures': 'w',\n 'no-wrap': 'w',\n 'number-offset': 'w',\n 'number-sections': 'w',\n 'pdf-engine': 'w',\n 'pdf-engine-opt': 'w',\n 'reference-doc': 'w',\n 'reference-links': 'w',\n 'reference-location': 'w',\n 'request-header': 'w',\n 'resource-path': 'w',\n 'section-divs': 'w',\n 'self-contained': 'w',\n 'slide-level': 'w',\n 'standalone': 'w',\n 'syntax-definition': 'w',\n 'table-of-contents': 'w',\n 'title-prefix': 'w',\n 'toc-depth': 'w',\n 'top-level-division': 'w',\n 'variable': 'w',\n 'verbose': 'w',\n 'webtex': 'w',\n 'wrap': 'w'\n}\n\n# Adapted from https://github.com/jgm/pandoc/blob/master/pandoc.hs#L841\nPANDOC_WRITER_MAPPING = {\n \"\" : \"markdown\",\n \".tex\" : \"latex\",\n \".latex\" : \"latex\",\n \".ltx\" : \"latex\",\n \".context\" : \"context\",\n \".ctx\" : \"context\",\n \".rtf\" : \"rtf\",\n \".rst\" : \"rst\",\n \".s5\" : \"s5\",\n \".native\" : \"native\",\n \".json\" : \"json\",\n \".txt\" : \"markdown\",\n \".text\" : \"markdown\",\n \".md\" : \"markdown\",\n \".markdown\" : \"markdown\",\n \".textile\" : \"textile\",\n \".lhs\" : \"markdown+lhs\",\n \".texi\" : \"texinfo\",\n \".texinfo\" : \"texinfo\",\n \".db\" : \"docbook\",\n \".odt\" : \"odt\",\n \".docx\" : \"docx\",\n \".epub\" : \"epub\",\n \".org\" : \"org\",\n \".asciidoc\" : \"asciidoc\",\n \".adoc\" : \"asciidoc\",\n \".fb2\" : \"fb2\",\n \".opml\" : \"opml\",\n \".icml\" : \"icml\",\n \".tei.xml\" : \"tei\",\n \".tei\" : \"tei\",\n \".ms\" : \"ms\",\n \".roff\" : \"ms\",\n \".pptx\" : \"pptx\",\n \".1\" : \"man\",\n \".2\" : \"man\",\n \".3\" : \"man\",\n \".4\" : \"man\",\n \".5\" : \"man\",\n \".6\" : \"man\",\n \".7\" : \"man\",\n \".8\" : \"man\",\n \".9\" : \"man\"\n}\n" }, { "path": "panzer/document.py", "content": "\"\"\" panzer document class and its methods \"\"\"\nimport json\nimport os\nimport pandocfilters\nimport subprocess\nimport sys\nfrom . import error\nfrom . import meta\nfrom . import util\nfrom . import info\nfrom . import const\n\nclass Document(object):\n \"\"\" representation of pandoc/panzer documents\n - ast: pandoc abstract syntax tree of document\n - style: list of styles for document\n - stylefull: full list of styles including all parents\n - styledef: style definitions\n - runlist: run list for document\n - options: panzer and pandoc command line options\n - template: template for document\n - output: string filled with output when processing complete\n \"\"\"\n #\n # disable pylint warnings:\n # + Too many instance attributes\n # pylint: disable=R0902\n #\n def __init__(self):\n \"\"\" new blank document \"\"\"\n # - defaults\n if const.USE_OLD_API:\n self.ast = const.EMPTY_DOCUMENT_OLDAPI\n else:\n self.ast = const.EMPTY_DOCUMENT\n self.style = list()\n self.stylefull = list()\n self.styledef = dict()\n self.runlist = list()\n self.template = None\n self.output = None\n self.options = {\n 'panzer': {\n 'panzer_support' : const.DEFAULT_SUPPORT_DIR,\n 'pandoc' : 'pandoc',\n 'debug' : str(),\n 'quiet' : False,\n 'strict' : False,\n 'stdin_temp_file' : str()\n },\n 'pandoc': {\n 'input' : ['-'],\n 'output' : '-',\n 'pdf_output' : False,\n 'read' : str(),\n 'write' : str(),\n 'template' : str(),\n 'filter' : list(),\n 'lua_filter' : list(),\n 'options' : {'r': dict(), 'w': dict()},\n 'mutable' : {'r': dict(), 'w': dict()}\n }\n }\n\n def empty(self):\n \"\"\"\n empty document of all its content but `self.options`\n (used when re-reading from input with new reader command line options)\n \"\"\"\n # - defaults\n if const.USE_OLD_API:\n self.ast = const.EMPTY_DOCUMENT_OLDAPI\n else:\n self.ast = const.EMPTY_DOCUMENT\n self.style = list()\n self.stylefull = list()\n self.styledef = dict()\n self.runlist = list()\n self.template = None\n self.output = None\n\n def populate(self, ast, global_styledef, local_styledef):\n \"\"\"\n populate document's:\n `self.ast`,\n `self.styledef`,\n `self.style`,\n `self.stylefull`\n remaining fields:\n `self.template` - set after 'transform' applied\n `self.runlist` - set after 'transform' applied\n `self.output` - set after 'pandoc' applied\n \"\"\"\n # - set self.ast:\n if ast:\n self.ast = ast\n else:\n info.log('ERROR', 'panzer', 'source document(s) empty')\n # - check if panzer_reserved key already exists in metadata\n metadata = self.get_metadata()\n try:\n meta.get_content(metadata, 'panzer_reserved')\n info.log('ERROR', 'panzer',\n 'special field \"panzer_reserved\" already in metadata'\n '---will be overwritten')\n except error.MissingField:\n pass\n # - set self.styledef\n self.populate_styledef(global_styledef, local_styledef)\n # - set self.style and self.stylefull\n self.populate_style()\n # - remove any styledef not used in doc\n self.styledef = {key: self.styledef[key]\n for key in self.styledef\n if key in self.stylefull}\n\n def populate_styledef(self, global_styledef, local_styledef):\n \"\"\"\n populate `self.styledef` from\n `global_styledef`\n `local_styledef`\n document style definition inside `styledef` metadata field\n \"\"\"\n info.log('INFO', 'panzer', info.pretty_title('style definitions'))\n # - print global style definitions\n if global_styledef:\n info.log('INFO', 'panzer', 'global:')\n for line in info.pretty_keys(global_styledef):\n info.log('INFO', 'panzer', ' ' + line)\n else:\n info.log('INFO', 'panzer', 'no global definitions loaded')\n # - print local style definitions\n overridden = dict()\n if local_styledef:\n info.log('INFO', 'panzer', 'local:')\n for line in info.pretty_keys(local_styledef):\n info.log('INFO', 'panzer', ' ' + line)\n # - extract and print document style definitions\n indoc_styledef = dict()\n try:\n indoc_styledef = meta.get_content(self.get_metadata(), 'styledef', 'MetaMap')\n info.log('INFO', 'panzer', 'document:')\n for line in info.pretty_keys(indoc_styledef):\n info.log('INFO', 'panzer', ' ' + line)\n except error.MissingField as err:\n info.log('DEBUG', 'panzer', err)\n except error.WrongType as err:\n info.log('ERROR', 'panzer', err)\n # - update the style definitions\n (self.styledef).update(global_styledef)\n (self.styledef).update(local_styledef)\n (self.styledef).update(indoc_styledef)\n # - print messages about overriding\n messages = list()\n messages += ['local document definition of \"%s\" overrides global definition of \"%s\"'\n % (key, key)\n for key in self.styledef\n if key in local_styledef\n and key in global_styledef]\n messages += ['document definition of \"%s\" overrides local definition of \"%s\"'\n % (key, key)\n for key in self.styledef\n if key in indoc_styledef\n and key in local_styledef]\n messages += ['document definition of \"%s\" overrides global definition of \"%s\"'\n % (key, key)\n for key in self.styledef\n if key in indoc_styledef\n and key in global_styledef\n and key not in local_styledef]\n for m in messages:\n info.log('INFO', 'panzer', m)\n\n def populate_style(self):\n \"\"\"\n populate `self.style` and `self.stylefull`\n \"\"\"\n info.log('INFO', 'panzer', info.pretty_title('document style'))\n # - try to extract value of style field\n try:\n self.style = meta.get_list_or_inline(self.get_metadata(), 'style')\n if self.style == ['']:\n raise error.MissingField\n except error.MissingField:\n info.log('INFO', 'panzer', 'no \"style\" field found, run only pandoc')\n return\n except error.WrongType as err:\n info.log('ERROR', 'panzer', err)\n return\n info.log('INFO', 'panzer', 'style:')\n info.log('INFO', 'panzer', info.pretty_list(self.style))\n # - expand the style hierarchy\n self.stylefull = meta.expand_style_hierarchy(self.style, self.styledef)\n info.log('INFO', 'panzer', 'full hierarchy:')\n info.log('INFO', 'panzer', info.pretty_list(self.stylefull))\n\n def build_runlist(self):\n \"\"\" populate `self.runlist` using `self.ast`'s metadata \"\"\"\n info.log('INFO', 'panzer', info.pretty_title('run list'))\n metadata = self.get_metadata()\n runlist = self.runlist\n for kind in const.RUNLIST_KIND:\n # - sanity check\n try:\n field_type = meta.get_type(metadata, kind)\n if field_type != 'MetaList':\n info.log('ERROR', 'panzer',\n 'value of field \"%s\" should be of type \"MetaList\"'\n '---found value of type \"%s\", ignoring it'\n % (kind, field_type))\n continue\n except error.MissingField:\n pass\n # - if 'filter', add filter list specified on command line first\n if kind == 'filter':\n for cmd in self.options['pandoc']['filter']:\n entry = dict()\n entry['kind'] = 'filter'\n entry['status'] = const.QUEUED\n entry['command'] = cmd[0]\n entry['arguments'] = list()\n runlist.append(entry)\n # - if 'lua-filter', add filter list specified on command line first\n elif kind == 'lua-filter':\n for cmd in self.options['pandoc']['lua_filter']:\n entry = dict()\n entry['kind'] = 'lua-filter'\n entry['status'] = const.QUEUED\n entry['command'] = cmd[0]\n entry['arguments'] = list()\n runlist.append(entry)\n # - add commands specified in metadata\n if kind in metadata:\n entries = meta.get_runlist(metadata, kind, self.options)\n runlist.extend(entries)\n # - now some cleanup:\n # -- filters: add writer as first argument\n for entry in runlist:\n if entry['kind'] == 'filter':\n writer = self.options['pandoc']['write']\n strip_exts = (writer.split('+')[0].split('-'))[0]\n entry['arguments'].insert(0, strip_exts)\n # -- postprocessors: remove them if output kind is pdf\n # .. or if a binary writer is selected\n if self.options['pandoc']['pdf_output'] \\\n or self.options['pandoc']['write'] in const.BINARY_WRITERS:\n new_runlist = list()\n for entry in runlist:\n if entry['kind'] == 'postprocess':\n info.log('INFO', 'panzer',\n 'postprocess \"%s\" skipped --- output of pandoc is binary file'\n % entry['command'])\n continue\n new_runlist.append(entry)\n runlist = new_runlist\n msg = info.pretty_runlist(runlist)\n for line in msg:\n info.log('INFO', 'panzer', line)\n self.runlist = runlist\n\n def apply_commandline(self, metadata):\n \"\"\"\n 1. parse `self.ast`'s `commandline` field\n 2. apply result to update self.options['pandoc']['options']\n (the result are the options used for calling pandoc)\n \"\"\"\n if 'commandline' not in metadata:\n return\n commandline = meta.parse_commandline(metadata)\n if not commandline:\n return\n self.options['pandoc']['options'] = \\\n meta.update_pandoc_options(self.options['pandoc']['options'],\n commandline,\n self.options['pandoc']['mutable'])\n\n def lock_commandline(self):\n \"\"\"\n make the commandline line options all immutable\n \"\"\"\n for phase in self.options['pandoc']['mutable']:\n for opt in self.options['pandoc']['mutable'][phase]:\n self.options['pandoc']['mutable'][phase][opt] = False\n\n def json_message(self, clear=False):\n \"\"\"\n create json message to pass to executables. This method does 2 things:\n\n 1. injects json message into `panzer_reserved` field of `self.ast`\n 2. returns json message as a string\n\n if 'clear' is set, then embedded json message is removeed and None returned\n \"\"\"\n metadata = self.get_metadata()\n # - delete old 'panzer_reserved' key\n if 'panzer_reserved' in metadata:\n del metadata['panzer_reserved']\n if clear:\n self.set_metadata(metadata)\n return None\n # - create a decrapified version of self.options\n # - remove stuff only of internal use to panzer\n options = dict()\n options['panzer'] = dict(self.options['panzer'])\n options['pandoc'] = dict(self.options['pandoc'])\n del options['pandoc']['template']\n del options['pandoc']['filter']\n del options['pandoc']['lua_filter']\n del options['pandoc']['mutable']\n # - build new json_message\n data = [{'metadata': metadata,\n 'template': self.template,\n 'style': self.style,\n 'stylefull': self.stylefull,\n 'styledef': self.styledef,\n 'runlist': self.runlist,\n 'options': options}]\n json_message = json.dumps(data)\n # - inject into metadata\n content = {\"json_message\": {\n \"t\": \"MetaBlocks\",\n \"c\": [{\"t\": \"CodeBlock\", \"c\": [[\"\", [\"json\"], []], json_message]}]}}\n meta.set_content(metadata, 'panzer_reserved', content, 'MetaMap')\n self.set_metadata(metadata)\n # - return json_message\n return json_message\n\n def purge_style_fields(self):\n \"\"\" remove metadata fields from `self.ast` used to call panzer \"\"\"\n kill_list = const.RUNLIST_KIND\n kill_list += ['style']\n kill_list += ['styledef']\n kill_list += ['template']\n kill_list += ['commandline']\n metadata = self.get_metadata()\n new_metadata = {key: metadata[key]\n for key in metadata\n if key not in kill_list}\n self.set_metadata(new_metadata)\n\n def get_metadata(self):\n \"\"\" return metadata branch of `self.ast` \"\"\"\n return meta.get_metadata(self.ast)\n\n def set_metadata(self, new_metadata):\n \"\"\" set metadata branch of `self.ast` to `new_metadata` \"\"\"\n if const.USE_OLD_API:\n try:\n self.ast[0]['unMeta'] = new_metadata\n except (IndexError, KeyError):\n self.ast = const.EMPTY_DOCUMENT_OLDAPI\n self.ast[0]['unMeta'] = new_metadata\n else:\n self.ast['meta'] = new_metadata\n\n def transform(self):\n \"\"\" transform `self` by applying styles listed in `self.stylefull` \"\"\"\n writer = self.options['pandoc']['write']\n info.log('INFO', 'panzer', 'writer:')\n info.log('INFO', 'panzer', ' %s' % writer)\n # 1. Do transform\n # - start with blank metadata\n new_metadata = dict()\n # - apply styles, first to last\n for style in self.stylefull:\n all_s = meta.get_nested_content(self.styledef, [style, 'all'], 'MetaMap')\n new_metadata = meta.update_metadata(new_metadata, all_s)\n self.apply_commandline(all_s)\n cur_s = meta.get_nested_content(self.styledef, [style, writer], 'MetaMap')\n new_metadata = meta.update_metadata(new_metadata, cur_s)\n self.apply_commandline(cur_s)\n # - add in document metadata in document\n indoc_data = self.get_metadata()\n # -- add items from additive fields in indoc metadata\n new_metadata = meta.update_additive_lists(new_metadata, indoc_data)\n for field in const.RUNLIST_KIND:\n if field in indoc_data:\n del indoc_data[field]\n # -- add all other (non-additive) fields in\n new_metadata.update(indoc_data)\n # -- apply items from indoc `commandline` field\n self.apply_commandline(indoc_data)\n # 2. Apply kill rules to trim run lists\n for field in const.RUNLIST_KIND:\n try:\n original_list = meta.get_content(new_metadata, field, 'MetaList')\n trimmed_list = meta.apply_kill_rules(original_list)\n if trimmed_list:\n meta.set_content(new_metadata, field, trimmed_list, 'MetaList')\n else:\n # if all items killed, delete field\n del new_metadata[field]\n except error.MissingField:\n continue\n except error.WrongType as err:\n info.log('WARNING', 'panzer', err)\n continue\n # 3. Set template\n try:\n if meta.get_type(new_metadata, 'template') == 'MetaInlines':\n template_raw = meta.get_content(new_metadata, 'template', 'MetaInlines')\n template_str = pandocfilters.stringify(template_raw)\n elif meta.get_type(new_metadata, 'template') == 'MetaString':\n template_str = meta.get_content(new_metadata, 'template', 'MetaString')\n if template_str == '':\n raise error.MissingField\n else:\n raise error.WrongType\n self.template = util.resolve_path(template_str, 'template', self.options)\n except (error.MissingField, error.WrongType) as err:\n info.log('DEBUG', 'panzer', err)\n if self.template:\n info.log('INFO', 'panzer', info.pretty_title('template'))\n info.log('INFO', 'panzer', ' %s' % info.pretty_path(self.template))\n # 4. Update document's metadata\n self.set_metadata(new_metadata)\n\n def run_scripts(self, kind, do_not_stop=False):\n \"\"\"\n execute commands of type `kind` listed in `self.runlist`\n `do_not_stop`: runlist executed no matter what errors occur\n (used by cleanup scripts)\n \"\"\"\n # - check if no run list to run\n to_run = [entry for entry in self.runlist if entry['kind'] == kind]\n if not to_run:\n return\n info.log('INFO', 'panzer', info.pretty_title(kind))\n # - maximum number of executables to run\n for i, entry in enumerate(self.runlist):\n # - skip entries that are not of the right kind\n if entry['kind'] != kind:\n continue\n # - build the command to run\n command = [entry['command']] + entry['arguments']\n filename = os.path.basename(entry['command'])\n info.log('INFO', 'panzer',\n info.pretty_runlist_entry(i,\n len(self.runlist),\n entry['command'],\n entry['arguments']))\n info.log('DEBUG', 'panzer', 'run \"%s\"' % ' '.join(command))\n # - run the command\n stderr = str()\n try:\n entry['status'] = const.RUNNING\n process = subprocess.Popen(' '.join(command),\n stdin=subprocess.PIPE,\n stderr=subprocess.PIPE,\n shell=True)\n # send panzer's json message to scripts via stdin\n in_pipe = self.json_message()\n in_pipe_bytes = in_pipe.encode(const.ENCODING)\n stderr_bytes = process.communicate(input=in_pipe_bytes)[1]\n entry['status'] = const.DONE\n stderr = stderr_bytes.decode(const.ENCODING)\n if stderr:\n entry['stderr'] = info.decode_stderr_json(stderr)\n except OSError as err:\n entry['status'] = const.FAILED\n info.log('ERROR', filename, err)\n continue\n except Exception as err: # pylint: disable=W0703\n # if do_not_stop: always run next script\n # disable pylint warnings:\n # + Catching too general exception\n entry['status'] = const.FAILED\n if do_not_stop:\n info.log('ERROR', filename, err)\n continue\n else:\n raise\n finally:\n info.log_stderr(stderr, filename)\n\n def jsonfilter(self):\n \"\"\"\n pipe through external command listed in filters\n \"\"\"\n to_run = [entry for entry in self.runlist if entry['kind'] == 'filter']\n if not to_run:\n return\n info.log('INFO', 'panzer', info.pretty_title('filter'))\n # Run commands\n for i, entry in enumerate(self.runlist):\n if entry['kind'] != 'filter':\n continue\n # - add debugging info\n command = [entry['command']] + entry['arguments']\n filename = os.path.basename(entry['command'])\n info.log('INFO', 'panzer',\n info.pretty_runlist_entry(i,\n len(self.runlist),\n entry['command'],\n entry['arguments']))\n info.log('DEBUG', 'panzer', 'run \"%s\"' % ' '.join(command))\n # - run the command and log any errors\n stderr = str()\n try:\n entry['status'] = const.RUNNING\n self.json_message()\n # Set up incoming pipe\n in_pipe = json.dumps(self.ast)\n # Set up outgoing pipe in case of failure\n out_pipe = in_pipe\n process = subprocess.Popen(' '.join(command),\n stderr=subprocess.PIPE,\n stdin=subprocess.PIPE,\n stdout=subprocess.PIPE,\n shell=True)\n in_pipe_bytes = in_pipe.encode(const.ENCODING)\n out_pipe_bytes, stderr_bytes = \\\n process.communicate(input=in_pipe_bytes)\n entry['status'] = const.DONE\n out_pipe = out_pipe_bytes.decode(const.ENCODING)\n stderr = stderr_bytes.decode(const.ENCODING)\n if stderr:\n entry['stderr'] = info.decode_stderr_json(stderr)\n except OSError as err:\n entry['status'] = const.FAILED\n info.log('ERROR', filename, err)\n continue\n except Exception:\n entry['status'] = const.FAILED\n raise\n finally:\n # remove embedded json message\n info.log_stderr(stderr, filename)\n # 4. Update document's data with output from commands\n try:\n self.ast = json.loads(out_pipe)\n self.json_message(clear=True)\n except ValueError:\n info.log('ERROR', 'panzer',\n 'failed to receive json object from filter'\n '---skipping filter')\n continue\n\n def pandoc(self):\n \"\"\"\n run pandoc on document\n\n Normally, input to pandoc is passed via stdin and output received via\n stout. Exception is when the output file has .pdf extension or a binary\n writer selected. Then, output is simply the binary file that panzer\n does not process further, and internal document not updated by pandoc.\n \"\"\"\n # 1. Build pandoc command\n command = [self.options['panzer']['pandoc']]\n command += ['-']\n command += ['--read', 'json']\n command += ['--write', self.options['pandoc']['write']]\n command += ['--output', self.options['pandoc']['output']]\n # - template specified on cli has precedence\n if self.options['pandoc']['template']:\n command += ['--template=%s' % self.options['pandoc']['template']]\n elif self.template:\n command += ['--template=%s' % self.template]\n opts = meta.build_cli_options(self.options['pandoc']['options']['w'])\n command += opts\n info.log('INFO', 'panzer', info.pretty_title('pandoc write'))\n # - add lua filters\n luaopts = list()\n for i, entry in enumerate(self.runlist):\n if entry['kind'] != 'lua-filter':\n continue\n command += ['--lua-filter', entry['command']]\n luaopts += ['--lua-filter', entry['command']]\n info.log('INFO', 'panzer',\n info.pretty_runlist_entry(i,\n len(self.runlist),\n entry['command'],\n entry['arguments']))\n # 2. Prefill input and output pipes\n in_pipe = json.dumps(self.ast)\n out_pipe = str()\n stderr = str()\n # 3. Run pandoc command\n if opts or luaopts:\n info.log('INFO', 'panzer', 'pandoc writing with options:')\n info.log('INFO', 'panzer', info.pretty_list(opts + luaopts, separator=' '))\n else:\n info.log('INFO', 'panzer', 'running')\n info.log('DEBUG', 'panzer', 'run \"%s\"' % ' '.join(command))\n try:\n info.time_stamp('ready to do popen')\n process = subprocess.Popen(command,\n stderr=subprocess.PIPE,\n stdin=subprocess.PIPE,\n stdout=subprocess.PIPE)\n info.time_stamp('popen done')\n in_pipe_bytes = in_pipe.encode(const.ENCODING)\n out_pipe_bytes, stderr_bytes = \\\n process.communicate(input=in_pipe_bytes)\n info.time_stamp('communicate done')\n out_pipe = out_pipe_bytes.decode(const.ENCODING)\n stderr = stderr_bytes.decode(const.ENCODING)\n except OSError as err:\n info.log('ERROR', 'pandoc', err)\n finally:\n info.log_stderr(stderr)\n sys.stdout.buffer.write(out_pipe_bytes)\n sys.stdout.flush()\n # mark all lua filters as 'done'\n for entry in self.runlist:\n if entry['kind'] == 'lua-filter':\n entry['status'] = const.DONE\n # 4. Capture output of pandoc if it is to stdout\n if self.options['pandoc']['pdf_output'] \\\n or self.options['pandoc']['write'] in const.BINARY_WRITERS:\n # do nothing with a binary output\n pass\n elif self.options['pandoc']['output'] == '-':\n self.output = out_pipe\n\n def postprocess(self):\n \"\"\"\n postprocess through external command listed in 'postprocess'\n \"\"\"\n to_run = [entry for entry in self.runlist if entry['kind'] == 'postprocess']\n if not to_run:\n return\n info.log('INFO', 'panzer', info.pretty_title('postprocess'))\n # prepare the input\n # case 1: pandoc output written to stdout\n if self.options['pandoc']['output'] == '-':\n in_pipe = self.output\n info.log('INFO', 'panzer', \"input read from pandoc's stdout\")\n # case 2: pandoc output written to file\n else:\n with open(self.options['pandoc']['output'], 'r', encoding=const.ENCODING) as fp:\n in_pipe = fp.read()\n info.log('INFO', 'panzer', 'input read from \"%s\"' % self.options['pandoc']['output'])\n # Run commands\n for i, entry in enumerate(self.runlist):\n if entry['kind'] != 'postprocess':\n continue\n # - add debugging info\n command = [entry['command']] + entry['arguments']\n filename = os.path.basename(entry['command'])\n info.log('INFO', 'panzer',\n info.pretty_runlist_entry(i,\n len(self.runlist),\n entry['command'],\n entry['arguments']))\n info.log('DEBUG', 'panzer', 'run \"%s\"' % ' '.join(command))\n # - run the command and log any errors\n stderr = str()\n try:\n entry['status'] = const.RUNNING\n # Set up outgoing pipe in case of failure\n out_pipe = in_pipe\n process = subprocess.Popen(' '.join(command),\n stderr=subprocess.PIPE,\n stdin=subprocess.PIPE,\n stdout=subprocess.PIPE,\n shell=True)\n in_pipe_bytes = in_pipe.encode(const.ENCODING)\n out_pipe_bytes, stderr_bytes = \\\n process.communicate(input=in_pipe_bytes)\n entry['status'] = const.DONE\n out_pipe = out_pipe_bytes.decode(const.ENCODING)\n stderr = stderr_bytes.decode(const.ENCODING)\n if stderr:\n entry['stderr'] = info.decode_stderr_json(stderr)\n except OSError as err:\n entry['status'] = const.FAILED\n info.log('ERROR', filename, err)\n continue\n except Exception:\n entry['status'] = const.FAILED\n raise\n finally:\n info.log_stderr(stderr, filename)\n self.output = out_pipe\n in_pipe = out_pipe\n # 4. write final output\n # case 1: stdout as output\n if self.options['pandoc']['output'] == '-':\n sys.stdout.buffer.write(out_pipe_bytes)\n sys.stdout.flush()\n info.log('INFO', 'panzer', 'output written to stdout')\n info.log('DEBUG', 'panzer', 'output written stdout by panzer')\n # case 2: output to file\n else:\n with open(self.options['pandoc']['output'], 'w',\n encoding=const.ENCODING) as output_file:\n output_file.write(out_pipe)\n output_file.flush()\n info.log('INFO', 'panzer', 'output written to \"%s\"'\n % self.options['pandoc']['output'])\n\n" }, { "path": "panzer/error.py", "content": "\"\"\" Exception classes for panzer \"\"\"\n\nclass PanzerError(Exception):\n \"\"\" base class for all panzer exceptions \"\"\"\n pass\n\nclass SetupError(PanzerError):\n \"\"\" error in the setup phase \"\"\"\n pass\n\nclass BadASTError(PanzerError):\n \"\"\" malformatted AST encountered (e.g. C or T fields missing) \"\"\"\n pass\n\nclass BadArgsFormat(PanzerError):\n \"\"\" args field for item in run list has incorrect format \"\"\"\n pass\n\nclass NoArgsAllowed(PanzerError):\n \"\"\" no command line arguments allowed to be passed to lua filters \"\"\"\n pass\n\nclass MissingField(PanzerError):\n \"\"\" looked for metadata field, did not find it \"\"\"\n pass\n\nclass WrongType(PanzerError):\n \"\"\" looked for value of a type, encountered different type \"\"\"\n pass\n\nclass InternalError(PanzerError):\n \"\"\" function invoked with invalid parameters \"\"\"\n pass\n\nclass StrictModeError(PanzerError):\n \"\"\"\n An error on `---strict` mode that causes panzer to exit\n - On `--strict` mode: exception raised if any error of level 'ERROR' or\n above is logged\n - Without `--strict` mode: exception never raised\n \"\"\"\n pass\n" }, { "path": "panzer/info.py", "content": "\"\"\" functions for logging and printing info \"\"\"\nimport json\nimport logging\nimport logging.config\nimport os\nimport time\nfrom . import const\nfrom . import error\n\n# - lookup table for internal strings to logging levels\nLEVELS = {\n 'CRITICAL' : logging.CRITICAL,\n 'ERROR' : logging.ERROR,\n 'WARNING' : logging.WARNING,\n 'INFO' : logging.INFO,\n 'DEBUG' : logging.DEBUG,\n 'NOTSET' : logging.NOTSET\n}\n\ndef start_logger(options):\n \"\"\" start the logger \"\"\"\n # - default configuration\n config = {\n 'version' : 1,\n 'disable_existing_loggers' : False,\n 'formatters': {\n 'detailed': {\n 'format': '%(asctime)s - %(levelname)s - %(message)s'\n },\n 'mimimal': {\n 'format': '%(message)s'\n }\n },\n 'handlers': {\n 'log_file_handler': {\n 'class' : 'logging.FileHandler',\n 'level' : 'DEBUG',\n 'formatter' : 'detailed',\n 'filename' : options['panzer']['debug'] + '.log',\n 'encoding' : const.ENCODING\n },\n 'console': {\n 'class' : 'logging.StreamHandler',\n 'level' : 'INFO',\n 'formatter' : 'mimimal',\n 'stream' : 'ext://sys.stderr'\n }\n },\n 'loggers': {\n __name__: {\n 'handlers' : ['console', 'log_file_handler'],\n 'level' : 'DEBUG',\n 'propagate' : True\n }\n }\n }\n # - set 'debug' mode if requested\n if not options['panzer']['debug']:\n config['loggers'][__name__]['handlers'].remove('log_file_handler')\n del config['handlers']['log_file_handler']\n else:\n # - delete old log file if it exists\n # - don't see value in keeping old logs here...\n filename = config['handlers']['log_file_handler']['filename']\n if os.path.exists(filename):\n os.remove(filename)\n # - set 'quiet' mode if requested\n if options['panzer']['quiet']:\n verbosity_level = 'WARNING'\n else:\n verbosity_level = 'INFO'\n config['handlers']['console']['level'] = verbosity_level\n # - set 'strict' mode if requested\n if options['panzer']['strict']:\n log.strict_mode = True\n # - send configuration to logger\n logging.config.dictConfig(config)\n log('DEBUG', 'panzer', pretty_start_log('panzer starts'))\n log('DEBUG', 'panzer', pretty_title('OPTIONS'))\n log('DEBUG', 'panzer', pretty_json_repr(options))\n\ndef log(level_str, sender, message):\n \"\"\" send a log message \"\"\"\n my_logger = logging.getLogger(__name__)\n # set strict_mode to default value, if not already set\n if not hasattr(log, \"strict_mode\"):\n log.strict_mode = False\n # - lookup table for internal strings to pretty output strings\n pretty_levels = {\n 'CRITICAL' : 'FATAL: ',\n 'ERROR' : 'ERROR: ',\n 'WARNING' : 'WARNING: ',\n 'INFO' : ' ',\n 'DEBUG' : ' ',\n 'NOTSET' : ' '\n }\n message = str(message)\n sender_str = ''\n message_str = ''\n level = LEVELS.get(level_str, LEVELS['ERROR'])\n # -- level\n pretty_level_str = pretty_levels.get(level_str, pretty_levels['ERROR'])\n # -- sender\n if sender != 'panzer':\n # sender_str = ' ' + sender + ': '\n sender_str = ' '\n # -- message\n message_str = message\n output = ''\n output += pretty_level_str\n output += sender_str\n output += message_str\n my_logger.log(level, output)\n # - if 'strict' mode and error logged, raise exception to exit panzer\n if log.strict_mode and (level_str == 'ERROR' or level_str == 'CRITICAL'):\n log.strict_mode = False\n raise error.StrictModeError\n\ndef go_quiet():\n \"\"\" force logging level to be --quiet \"\"\"\n my_logger = logging.getLogger(__name__)\n my_logger.setLevel(LEVELS['WARNING'])\n\ndef go_loud(options):\n \"\"\" return logging level to that set in options \"\"\"\n my_logger = logging.getLogger(__name__)\n if options['panzer']['quiet']:\n verbosity_level = 'WARNING'\n else:\n verbosity_level = 'INFO'\n my_logger.setLevel(LEVELS[verbosity_level])\n\ndef decode_stderr_json(stderr):\n \"\"\" return a list of decoded json messages in stderr \"\"\"\n # - check for blank input\n if not stderr:\n # - nothing to do\n return list()\n # - split the input (based on newlines) into list of json strings\n output = list()\n for line in stderr.split('\\n'):\n if not line:\n # - skip blank lines: no valid json or message to decode\n continue\n json_message = list()\n try:\n json_message = json.loads(line)\n except ValueError:\n # - if json cannot be decoded, just log as ERROR prefixed by '!'\n json_message = {'level': 'ERROR', 'message': '!' + line}\n output.append(json_message)\n return output\n\ndef log_stderr(stderr, sender=str()):\n \"\"\" send a log from external executable \"\"\"\n # 1. check for blank input\n if not stderr:\n # - nothing to do\n return\n # 2. get a string with sender's name\n if sender:\n # - remove file extension from sender's name if present\n sender = os.path.splitext(sender)[0]\n # 3. now handle the messages sent by sender\n json_message = decode_stderr_json(stderr)\n for item in json_message:\n level = item['level']\n message = item['message']\n log(level, sender, message)\n\ndef pretty_keys(dictionary):\n \"\"\" return pretty printed list of dictionary keys, num per line \"\"\"\n if not dictionary:\n return []\n # - number of keys printed per line\n num = 5\n # - turn into sorted list\n keys = list(dictionary.keys())\n keys.sort()\n # - fill with blank elements to width num\n missing = (len(keys) % num)\n if missing != 0:\n to_add = num - missing\n keys.extend([''] * to_add)\n # - turn into 2D matrix\n matrix = [[keys[i+j] for i in range(0, num)]\n for j in range(0, len(keys), num)]\n # - calculate max width for each column\n len_matrix = [[len(col) for col in row] for row in matrix]\n max_len_col = [max([row[j] for row in len_matrix])\n for j in range(0, num)]\n # - pad with spaces\n matrix = [[row[j].ljust(max_len_col[j]) for j in range(0, num)]\n for row in matrix]\n # - return list of lines to print\n matrix = [' '.join(row) for row in matrix]\n return matrix\n\ndef pretty_list(input_list, separator=', '):\n \"\"\" return pretty printed list \"\"\"\n if input_list:\n output = ' %s' % separator.join(input_list)\n else:\n output = ' empty'\n return output\n\ndef pretty_json_repr(data):\n \"\"\" return pretty printed data as a json \"\"\"\n return json.dumps(data, sort_keys=True, indent=2)\n\ndef pretty_title(title):\n \"\"\" return pretty printed section title \"\"\"\n output = '-' * 5 + ' ' + title.lower() + ' ' + '-' * 5\n return output\n\ndef pretty_start_log(title):\n \"\"\" return pretty printed title for starting log \"\"\"\n output = '>' * 10 + ' ' + title + ' ' + '<' * 10\n return output\n\ndef pretty_end_log(title):\n \"\"\" return pretty printed title for ending log \"\"\"\n output = '>' * 10 + ' ' + title + ' ' + '<' * 10 + '\\n\\n'\n return output\n\ndef pretty_path(input_path):\n \"\"\" return path string replacing '~' for home directory \"\"\"\n home_path = os.path.expanduser('~')\n cwd_path = os.getcwd()\n output_path = input_path.replace(home_path, '~').replace(cwd_path, './')\n return output_path\n\ndef pretty_runlist(runlist):\n \"\"\" return pretty printed runlist \"\"\"\n if not runlist:\n return [' empty']\n output = list()\n current_kind = str()\n for i, entry in enumerate(runlist):\n if current_kind != entry['kind']:\n output.append(entry['kind'] + ':')\n current_kind = entry['kind']\n basename = pretty_path(entry['command'])\n if entry['arguments']:\n basename += ' '\n basename += ' '.join(entry['arguments'])\n line = '%d' % (i+1)\n line = line.rjust(3, ' ')\n line += ' %s' % basename\n output.append(line)\n return output\n\ndef pretty_runlist_entry(num, max_num, command, arguments):\n \"\"\" return pretty printed run list entry \"\"\"\n basename = command\n if arguments:\n basename += ' '\n basename += ' '.join(arguments)\n cur = '%d' % (num+1)\n cur = cur.rjust(len(str(max_num)), ' ')\n line = ' [%s/%d] %s' % (cur, max_num, basename)\n return line\n\ndef time_stamp(text):\n \"\"\"\n print time since first & previous time_stamp call\n \"\"\"\n if not const.DEBUG_TIMING:\n return\n try:\n now = time.time() - time_stamp.start\n except AttributeError:\n time_stamp.start = time.time()\n now = 0\n try:\n elapsed = now - time_stamp.last\n except AttributeError:\n elapsed = 0\n now_str = str(round(now * 1000)).rjust(7)\n now_str += ' msec'\n now_str += ' '\n now_str += text.ljust(30)\n if elapsed * 1000 > 1:\n now_str += str(round(elapsed * 1000)).rjust(7)\n now_str += ' msec'\n else:\n now_str += ' ' * 12\n time_stamp.last = now\n print(now_str)\n\n" }, { "path": "panzer/load.py", "content": "\"\"\" loading documents into panzer \"\"\"\n\nimport os\nimport json\nimport subprocess\nfrom . import error\nfrom . import info\nfrom . import const\nfrom . import meta\n\ndef load(options):\n \"\"\" return ast from running pandoc on input documents \"\"\"\n # 1. Build pandoc command\n command = [options['panzer']['pandoc']]\n command += options['pandoc']['input'].copy()\n if options['pandoc']['read']:\n command += ['--read', options['pandoc']['read']]\n command += ['--write', 'json', '--output', '-']\n opts = meta.build_cli_options(options['pandoc']['options']['r'])\n command += opts\n info.log('INFO', 'panzer', info.pretty_title('pandoc read'))\n info.log('DEBUG', 'panzer', 'loading source document(s)')\n info.log('DEBUG', 'panzer', 'run \"%s\"' % ' '.join(command))\n if opts:\n info.log('INFO', 'panzer', 'pandoc reading with options:')\n info.log('INFO', 'panzer', info.pretty_list(opts, separator=' '))\n else:\n info.log('INFO', 'panzer', 'running')\n out_pipe = str()\n stderr = str()\n ast = None\n try:\n process = subprocess.Popen(command,\n stderr=subprocess.PIPE,\n stdout=subprocess.PIPE)\n out_pipe_bytes, stderr_bytes = process.communicate()\n out_pipe = out_pipe_bytes.decode(const.ENCODING)\n stderr = stderr_bytes.decode(const.ENCODING)\n except OSError as err:\n info.log('ERROR', 'pandoc', err)\n finally:\n info.log_stderr(stderr)\n try:\n ast = json.loads(out_pipe)\n except ValueError:\n raise error.BadASTError('failed to receive valid '\n 'json object from pandoc')\n return ast\n\ndef load_all_styledefs(options):\n \"\"\"\n return global, local styledef pair\n finds global styledef from `.panzer/styles/*.{yaml,yml}`\n finds local styledef from `./styles/*.{yaml,yml}`\n \"\"\"\n support_dir = options['panzer']['panzer_support']\n info.log('DEBUG', 'panzer', 'loading global style definitions file')\n global_styledef = load_styledef(support_dir, options)\n if global_styledef == {}:\n info.log('WARNING', 'panzer', 'no global style definitions found')\n info.log('DEBUG', 'panzer', 'loading local style definitions file')\n local_styledef = load_styledef('.', options)\n return global_styledef, local_styledef\n\ndef load_styledef(path, options):\n \"\"\"\n return metadata branch as dict of styledef file at `path`\n reads from `path/styles/*.{yaml,yaml}`\n (if this fails, checks `path/styles.yaml` as legacy option)\n returns {} if no metadata found\n \"\"\"\n # - read in style definition data from yaml files\n styles_dir = os.path.join(path, 'styles')\n filenames = list()\n # - read from .panzer/styles/*.{yaml,yml}\n if os.path.exists(styles_dir):\n filenames = [os.path.join(path, 'styles', f)\n for f in os.listdir(styles_dir)\n if f.endswith('.yaml')\n or f.endswith('.yml')]\n # - read .panzer/styles.yaml -- legacy option\n elif os.path.exists(os.path.join(path, 'styles.yaml')):\n filenames = [os.path.join(path, 'styles.yaml')]\n data = list()\n for f in filenames:\n with open(f, 'r', encoding=const.ENCODING) as styles_file:\n data += styles_file.readlines()\n data += ['\\n']\n if data == []:\n return dict()\n # - top and tail with metadata markings\n data.insert(0, \"---\\n\")\n data.append(\"...\\n\")\n data_string = ''.join(data)\n # - build pandoc command\n command = [options['panzer']['pandoc']]\n command += ['-']\n command += ['--write', 'json']\n command += ['--output', '-']\n opts = meta.build_cli_options(options['pandoc']['options']['r'])\n # - remove inappropriate options for styles.yaml\n BAD_OPTS = ['metadata', 'track-changes', 'extract-media']\n opts = [x for x in opts if x not in BAD_OPTS]\n command += opts\n info.log('DEBUG', 'panzer', 'run \"%s\"' % ' '.join(command))\n # - send to pandoc to convert to json\n in_pipe = data_string\n out_pipe = ''\n stderr = ''\n try:\n process = subprocess.Popen(command,\n stderr=subprocess.PIPE,\n stdin=subprocess.PIPE,\n stdout=subprocess.PIPE)\n in_pipe_bytes = in_pipe.encode(const.ENCODING)\n out_pipe_bytes, stderr_bytes = process.communicate(input=in_pipe_bytes)\n out_pipe = out_pipe_bytes.decode(const.ENCODING)\n stderr = stderr_bytes.decode(const.ENCODING)\n except OSError as err:\n info.log('ERROR', 'pandoc', err)\n finally:\n info.log_stderr(stderr)\n # - convert json to python dict\n ast = None\n try:\n ast = json.loads(out_pipe)\n except ValueError:\n raise error.BadASTError('failed to receive valid '\n 'json object from pandoc')\n # - return metadata branch of dict\n if not ast:\n return dict()\n else:\n return meta.get_metadata(ast)\n\n" }, { "path": "panzer/meta.py", "content": "\"\"\" Functions for manipulating metadata \"\"\"\nimport pandocfilters\nimport shlex\nfrom . import const\nfrom . import info\nfrom . import util\nfrom . import error\n\ndef update_metadata(old, new):\n \"\"\" return `old` updated with `new` metadata \"\"\"\n # 1. Update with values in 'metadata' field\n try:\n old.update(get_content(new, 'metadata', 'MetaMap'))\n except (error.MissingField, KeyError):\n pass\n except error.WrongType as err:\n info.log('WARNING', 'panzer', err)\n # 2. Update with values in fields for additive lists\n old = update_additive_lists(old, new)\n # 3. Update 'template' field\n if 'template' in new:\n old['template'] = new['template']\n return old\n\ndef update_additive_lists(old, new):\n \"\"\" return old updated with info from additive lists in new \"\"\"\n for field in const.RUNLIST_KIND:\n try:\n try:\n new_list = get_content(new, field, 'MetaList')\n except error.MissingField:\n # field not in incoming metadata, move to next list\n continue\n try:\n old_list = get_content(old, field, 'MetaList')\n except error.MissingField:\n # field not in old metadata, start with an empty list\n old_list = list()\n except error.WrongType as err:\n # wrong type of value under field, skip to next list\n info.log('WARNING', 'panzer', err)\n continue\n old_list.extend(new_list)\n set_content(old, field, old_list, 'MetaList')\n return old\n\ndef apply_kill_rules(old_list):\n \"\"\" return old_list after applying kill rules \"\"\"\n new_list = list()\n for item in old_list:\n # 1. Sanity checks\n check_c_and_t_exist(item)\n item_content = item[const.C]\n item_type = item[const.T]\n if item_type != 'MetaMap':\n info.log('ERROR', 'panzer',\n 'fields \"' + '\", \"'.join(const.RUNLIST_KIND) + '\" '\n 'value must be of type \"MetaMap\"---ignoring 1 item')\n continue\n if len(item_content.keys() & {'run', 'kill', 'killall'}) != 1:\n info.log('ERROR', 'panzer',\n 'must contain exactly one \"run\", \"kill\", '\n 'or \"killall\" per item---ignoring 1 item')\n continue\n # 2. Now operate on content\n if 'run' in item_content:\n if get_type(item_content, 'run') != 'MetaInlines':\n info.log('ERROR', 'panzer',\n '\"run\" value must be of type \"MetaInlines\"'\n '---ignoring 1 item')\n continue\n new_list.append(item)\n elif 'kill' in item_content:\n try:\n to_be_killed = get_content(item_content, 'kill', 'MetaInlines')\n except error.WrongType as err:\n info.log('WARNING', 'panzer', err)\n continue\n new_list = [i for i in new_list\n if get_content(i[const.C],\n 'run',\n 'MetaInlines') != to_be_killed]\n continue\n elif 'killall' in item_content:\n try:\n if get_content(item_content, 'killall', 'MetaBool') is True:\n new_list = list()\n except error.WrongType as err:\n info.log('WARNING', 'panzer', err)\n continue\n else:\n # Should never occur, caught by previous syntax check\n continue\n return new_list\n\ndef get_nested_content(metadata, fields, expected_type_of_leaf=None):\n \"\"\" return content of field by traversing a list of MetaMaps\n\n args:\n metadata : dictionary to traverse\n fields : list of fields to traverse in dictionary from\n shallowest to deepest. Content of every field, except the last,\n must be type 'MetaMap' (otherwise fields could not be traversed).\n The content of final field in the list is returned.\n expected_type_of_leaf : (optional) expected type of final field's\n content\n\n Returns:\n content of final field in list, or the empty dict ({}) if field of\n expected type is not found\n \"\"\"\n current_field = fields.pop(0)\n try:\n # If on a branch...\n if fields:\n next_content = get_content(metadata, current_field, 'MetaMap')\n return get_nested_content(next_content, fields,\n expected_type_of_leaf)\n # Else on a leaf...\n else:\n return get_content(metadata, current_field, expected_type_of_leaf)\n except error.MissingField:\n # current_field not found, return {}: nothing to update\n return dict()\n except error.WrongType as err:\n info.log('WARNING', 'panzer', err)\n # wrong type found, return {}: nothing to update\n return dict()\n\ndef get_content(metadata, field, expected_type=None):\n \"\"\" return content of field \"\"\"\n if field not in metadata:\n raise error.MissingField('field \"%s\" not found' % field)\n check_c_and_t_exist(metadata[field])\n if expected_type:\n found_type = metadata[field][const.T]\n if found_type != expected_type:\n raise error.WrongType('value of \"%s\": expecting type \"%s\", '\n 'but found type \"%s\"'\n % (field, expected_type, found_type))\n return metadata[field][const.C]\n\ndef get_type(metadata, field):\n \"\"\" return type of field \"\"\"\n if field not in metadata:\n raise error.MissingField('field \"%s\" not found' % field)\n check_c_and_t_exist(metadata[field])\n return metadata[field][const.T]\n\ndef set_content(metadata, field, content, content_type):\n \"\"\" set content and type of field in metadata \"\"\"\n metadata[field] = {const.C: content, const.T: content_type}\n\ndef get_list_or_inline(metadata, field):\n \"\"\" return content of MetaList or MetaInlines item coerced as list \"\"\"\n field_type = get_type(metadata, field)\n if field_type == 'MetaInlines':\n content_raw = get_content(metadata, field, 'MetaInlines')\n content = [pandocfilters.stringify(content_raw)]\n return content\n elif field_type == 'MetaString':\n content_raw = get_content(metadata, field, 'MetaString')\n content = [content_raw]\n return content\n elif field_type == 'MetaList':\n content = list()\n for content_raw in get_content(metadata, field, 'MetaList'):\n content.append(pandocfilters.stringify(content_raw))\n return content\n else:\n raise error.WrongType('\"%s\" value must be of type \"MetaInlines\", '\n '\"MetaList\", or \"MetaString\"' % field)\n\ndef get_metadata(ast):\n \"\"\" returns metadata branch of ast or {} if not present \"\"\"\n try:\n if const.USE_OLD_API:\n metadata = ast[0]['unMeta']\n else:\n metadata = ast['meta']\n except KeyError:\n metadata = dict()\n return metadata\n\ndef get_runlist(metadata, kind, options):\n \"\"\" return run list for kind from metadata \"\"\"\n runlist = list()\n # - return empty list unless entries of kind are in metadata\n try:\n metadata_list = get_content(metadata, kind, 'MetaList')\n except (error.WrongType, error.MissingField) as err:\n info.log('WARNING', 'panzer', err)\n return runlist\n for item in metadata_list:\n check_c_and_t_exist(item)\n item_content = item[const.C]\n # - create new entry\n entry = dict()\n entry['kind'] = kind\n entry['command'] = str()\n entry['status'] = const.QUEUED\n # - get entry command\n command_raw = get_content(item_content, 'run', 'MetaInlines')\n command_str = pandocfilters.stringify(command_raw)\n entry['command'] = util.resolve_path(command_str, kind, options)\n # - get entry arguments\n entry['arguments'] = list()\n if 'args' in item_content:\n try:\n # - lua filters cannot take arguments\n if kind == 'lua-filter':\n raise error.NoArgsAllowed\n if get_type(item_content, 'args') != 'MetaInlines':\n raise error.BadArgsFormat\n args_content = get_content(item_content, 'args', 'MetaInlines')\n if len(args_content) != 1 \\\n or args_content[0][const.T] != 'Code':\n raise error.BadArgsFormat\n arguments_raw = args_content[0][const.C][1]\n entry['arguments'] = shlex.split(arguments_raw)\n except error.NoArgsAllowed:\n info.log('ERROR', 'panzer', '\"%s\": lua filters do not take arguments -- arguments ignored' %\n command_str)\n entry['arguments'] = list()\n except error.BadArgsFormat:\n info.log('ERROR', 'panzer', 'Cannot read \"args\" of \"%s\". '\n 'Syntax should be args: \"`--ARGUMENTS`\"'\n % command_str)\n entry['arguments'] = list()\n runlist.append(entry)\n return runlist\n\ndef check_c_and_t_exist(item):\n \"\"\" check item contains both C and T fields \"\"\"\n if const.C not in item:\n message = 'Value of \"%s\" corrupt: \"C\" field missing' % repr(item)\n raise error.BadASTError(message)\n if const.T not in item:\n message = 'Value of \"%s\" corrupt: \"T\" field missing' % repr(item)\n raise error.BadASTError(message)\n\ndef expand_style_hierarchy(stylelist, styledef):\n \"\"\" return stylelist expanded to include all parent styles \"\"\"\n expanded_list = []\n for style in stylelist:\n if style not in styledef:\n # - style not in styledef tree\n info.log('ERROR', 'panzer',\n 'No style definition found for style \"%s\" --- ignoring it'\n % style)\n continue\n defcontent = get_content(styledef, style, 'MetaMap')\n if 'parent' in defcontent:\n # - non-leaf node\n parents = get_list_or_inline(defcontent, 'parent')\n expanded_list.extend(expand_style_hierarchy(parents, styledef))\n expanded_list.append(style)\n return expanded_list\n\ndef build_cli_options(dic):\n \"\"\"\n return a sorted list of command line options specified in the options\n dictionary `dic`\n \"\"\"\n # - flags\n flags = ['--%s' % opt for opt in dic\n if dic[opt] == True]\n flags.sort()\n # - key-values\n keyvals = ['--%s=%s' % (opt, dic[opt]) for opt in dic\n if type(dic[opt]) is str]\n keyvals.sort()\n # - repeated key-values\n rkeys = [key for key in dic if type(dic[key]) is list]\n rkeys.sort()\n rkeyvals = list()\n for key in rkeys:\n rkeyvals += ['--%s=%s' % (key, val[0]) for val in dic[key]]\n return flags + keyvals + rkeyvals\n\ndef parse_commandline(metadata):\n \"\"\" return a dictiory of pandoc command line options by parsing\n `commandline` field in metadata; return None if `commandline` is absent in\n metadata\n \"\"\"\n if 'commandline' not in metadata:\n return None\n field_type = get_type(metadata, 'commandline')\n if field_type != 'MetaMap':\n info.log('ERROR', 'panzer',\n 'Value of field \"%s\" should be of type \"MetaMap\"'\n '---found value of type \"%s\", ignoring it'\n % ('commandline', field_type))\n return None\n content = get_content(metadata, 'commandline')\n # 1. remove bad options from `commandline`\n bad_opts = list(const.PANDOC_BAD_COMMANDLINE)\n for key in content:\n if key in bad_opts:\n info.log('ERROR', 'panzer',\n '\"%s\" forbidden entry in panzer \"commandline\" '\n 'map---ignoring' % key)\n if key not in const.PANDOC_OPT_PHASE:\n info.log('ERROR', 'panzer',\n 'do not recognise pandoc command line option \"--%s\" in \"commandline\" '\n 'map---ignoring' % key)\n bad_opts += key\n content = {key: content[key]\n for key in content\n if key not in bad_opts}\n # 2. parse remaining opts\n commandline = {'r': dict(), 'w': dict()}\n for key in content:\n # 1. extract value of field with name 'key'\n val = None\n val_t = get_type(content, key)\n val_c = get_content(content, key)\n # if value is 'false', set OPTION: False\n if val_t == 'MetaBool' and val_c is False:\n val = False\n # if value is 'true', set OPTION: True\n elif val_t == 'MetaBool' and val_c is True \\\n and key not in const.PANDOC_OPT_ADDITIVE:\n val = True\n # if value type is inline code, set OPTION: VAL\n elif val_t == 'MetaInlines':\n if len(val_c) != 1 or val_c[0][const.T] != 'Code':\n info.log('ERROR', 'panzer',\n 'Cannot read option \"%s\" in \"commandline\" field. '\n 'Syntax should be OPTION: \"`VALUE`\"' % key)\n continue\n if key in const.PANDOC_OPT_ADDITIVE:\n val = [get_list_or_inline(content, key)]\n else:\n val = get_list_or_inline(content, key)[0]\n # if value type is list of inline codes, set OPTION: [VALS]\n elif val_t == 'MetaList' and key in const.PANDOC_OPT_ADDITIVE:\n errs = False\n for item in val_c:\n if item[const.T] != 'MetaInlines' \\\n or item[const.C][0][const.T] != 'Code':\n info.log('ERROR', 'panzer',\n 'Cannot read option \"%s\" in \"commandline\" field. '\n 'Syntax should be - OPTION: \"`VALUE`\"' % key)\n errs = True\n if not errs:\n val = [[x] for x in get_list_or_inline(content, key)]\n else:\n continue\n # otherwise, signal error\n else:\n info.log('ERROR', 'panzer',\n 'Cannot read entry \"%s\" with type \"%s\" in '\n '\"commandline\"---ignoring' % (key, val_t))\n continue\n # 2. update commandline dictionary with key, val\n for phase in const.PANDOC_OPT_PHASE[key]:\n commandline[phase][key] = val\n return commandline\n\ndef update_pandoc_options(old, new, mutable):\n \"\"\"\n return dictionary of pandoc command line options 'old' updated with 'new'\n only options marked as mutable can be changed\n \"\"\"\n for p in ['r', 'w']:\n for key in new[p]:\n # if not mutable commandline line option, then skip it\n if not mutable[p][key]:\n continue\n # if 'False', reset old[p][key] to default\n elif new[p][key] is False:\n if type(old[p][key]) is list:\n old[p][key] = list()\n elif type(old[p][key]) is str:\n old[p][key] = None\n elif type(old[p][key]) is bool:\n old[p][key] = False\n # if list, extend old list with new\n elif key in old[p] and type(old[p][key]) is list:\n old[p][key].extend(new[p][key])\n # otherwise, override old with new\n else:\n old[p][key] = new[p][key]\n return old\n\n" }, { "path": "panzer/panzer.py", "content": "#!/usr/bin/env python3\n\"\"\" panzer: pandoc with styles\n\nfor more info: \n\nAuthor : Mark Sprevak \nCopyright : Copyright 2015, Mark Sprevak\nLicense : BSD3\n\"\"\"\n\nimport json\nimport os\nimport subprocess\nimport sys\nfrom . import cli\nfrom . import document\nfrom . import error\nfrom . import info\nfrom . import load\nfrom . import meta\nfrom . import util\nfrom . import version\n\n__version__ = version.VERSION\n\n# Main function\n\ndef main():\n \"\"\" the main function \"\"\"\n info.time_stamp('panzer started')\n doc = document.Document()\n try:\n doc.options = cli.parse_cli_options(doc.options)\n util.check_pandoc_exists(doc.options)\n old_reader_opts = dict(doc.options['pandoc']['options']['r'])\n info.time_stamp('cli options parsed')\n info.start_logger(doc.options)\n info.time_stamp('logger started')\n util.check_support_directory(doc.options)\n info.time_stamp('support directory checked')\n global_styledef, local_styledef = load.load_all_styledefs(doc.options)\n info.time_stamp('local + global styledefs loaded')\n ast = load.load(doc.options)\n info.time_stamp('document loaded')\n doc.populate(ast, global_styledef, local_styledef)\n doc.transform()\n doc.lock_commandline()\n new_reader_opts = doc.options['pandoc']['options']['r']\n # check if `commandline` contains any new reader options\n if new_reader_opts != old_reader_opts:\n # re-read input documents with new reader settings\n opts = meta.build_cli_options(new_reader_opts)\n info.log('INFO', 'panzer', info.pretty_title('pandoc read with metadata options'))\n info.log('INFO', 'panzer', 'pandoc reading with options:')\n info.log('INFO', 'panzer', info.pretty_list(opts, separator=' '))\n info.go_quiet()\n doc.empty()\n global_styledef, local_styledef = load.load_all_styledefs(doc.options)\n ast = load.load(doc.options)\n doc.populate(ast, global_styledef, local_styledef)\n doc.transform()\n info.go_loud(doc.options)\n doc.build_runlist()\n doc.purge_style_fields()\n info.time_stamp('document transformed')\n doc.run_scripts('preflight')\n info.time_stamp('preflight scripts done')\n doc.jsonfilter()\n info.time_stamp('json filters done')\n doc.pandoc()\n info.time_stamp('pandoc done')\n doc.postprocess()\n info.time_stamp('postprocess done')\n doc.run_scripts('postflight')\n info.time_stamp('postflight scripts done')\n except error.SetupError as err:\n # - errors that occur before logging starts\n print(err, file=sys.stderr)\n sys.exit(1)\n except error.StrictModeError:\n info.log('CRITICAL', 'panzer',\n 'cannot continue because error occurred while in \"strict\" mode')\n sys.exit(1)\n except subprocess.CalledProcessError:\n info.log('CRITICAL', 'panzer',\n 'cannot continue because of fatal error')\n sys.exit(1)\n except (KeyError,\n error.MissingField,\n error.BadASTError,\n error.WrongType,\n error.InternalError) as err:\n # - panzer exceptions not caught elsewhere, should have been\n info.log('CRITICAL', 'panzer', err)\n sys.exit(1)\n finally:\n doc.run_scripts('cleanup', do_not_stop=True)\n # - if temp file created in setup, remove it\n if doc.options['panzer']['stdin_temp_file']:\n os.remove(doc.options['panzer']['stdin_temp_file'])\n info.log('DEBUG', 'panzer', 'deleted temp file: %s'\n % doc.options['panzer']['stdin_temp_file'])\n # - write json message to file if ---debug set\n if doc.options['panzer']['debug']:\n filename = doc.options['panzer']['debug'] + '.json'\n content = info.pretty_json_repr(json.loads(doc.json_message()))\n with open(filename, 'w', encoding='utf8') as output_file:\n output_file.write(content)\n output_file.flush()\n info.log('DEBUG', 'panzer', info.pretty_end_log('panzer quits'))\n\n # - successful exit\n info.time_stamp('finished')\n sys.exit(0)\n\nif __name__ == '__main__':\n main()\n" }, { "path": "panzer/util.py", "content": "\"\"\" Support functions for non-core operations \"\"\"\nimport os\nimport subprocess\nimport sys\nfrom . import const\nfrom . import error\nfrom . import info\n\ndef check_pandoc_exists(options):\n \"\"\" check pandoc exists \"\"\"\n try:\n stdout_bytes = subprocess.check_output([options['panzer']['pandoc'],\n \"--version\"])\n stdout = stdout_bytes.decode(const.ENCODING)\n except PermissionError as err:\n raise error.SetupError('%s cannot be executed as pandoc executable' %\n options['panzer']['pandoc'])\n except OSError as err:\n if err.errno == os.errno.ENOENT:\n raise error.SetupError('%s not found as pandoc executable' %\n options['panzer']['pandoc'])\n else:\n raise error.SetupError(err)\n stdout_list = stdout.splitlines()\n pandoc_ver = stdout_list[0].split(' ')[1]\n # print('pandoc version: %s' % pandoc_ver, file=sys.stderr)\n if versiontuple(pandoc_ver) < versiontuple(const.REQUIRE_PANDOC_ATLEAST):\n raise error.SetupError('pandoc %s or greater required'\n '---found pandoc version %s'\n % (const.REQUIRE_PANDOC_ATLEAST, pandoc_ver))\n # check whether to use the new >=1.18 pandoc API or old (<1.18) one\n NEW_PANDOC_API = \"1.18\"\n if versiontuple(pandoc_ver) < versiontuple(NEW_PANDOC_API):\n const.USE_OLD_API = True\n # print('using old (<1.18) pandoc API')\n # else:\n # print('using new (>=1.18) pandoc API')\n\ndef versiontuple(version_string):\n \"\"\" return tuple of version_string \"\"\"\n # pylint: disable=W0141\n # disable warning for using builtin 'map'\n return tuple(map(int, (version_string.split(\".\"))))\n\ndef check_support_directory(options):\n \"\"\" check support directory exists \"\"\"\n if options['panzer']['panzer_support'] != const.DEFAULT_SUPPORT_DIR:\n if not os.path.exists(options['panzer']['panzer_support']):\n info.log('ERROR', 'panzer',\n 'panzer support directory \"%s\" not found'\n % options['panzer']['panzer_support'])\n info.log('WARNING', 'panzer',\n 'using default panzer support directory: %s'\n % const.DEFAULT_SUPPORT_DIR)\n options['panzer']['panzer_support'] = const.DEFAULT_SUPPORT_DIR\n if options['panzer']['panzer_support'] == const.DEFAULT_SUPPORT_DIR:\n if not os.path.exists(const.DEFAULT_SUPPORT_DIR):\n info.log('WARNING', 'panzer',\n 'default panzer support directory \"%s\" not found'\n % const.DEFAULT_SUPPORT_DIR)\n info.log('WARNING', 'panzer',\n 'create empty support directory \"%s\"?'\n % const.DEFAULT_SUPPORT_DIR)\n input(\" Press Enter to continue...\")\n create_default_support_dir()\n os.environ['PANZER_SHARED'] = \\\n os.path.join(options['panzer']['panzer_support'], 'shared')\n\ndef create_default_support_dir():\n \"\"\" create a empty panzer support directory \"\"\"\n # - create .panzer\n os.mkdir(const.DEFAULT_SUPPORT_DIR)\n info.log('INFO', 'panzer', 'created \"%s\"' % const.DEFAULT_SUPPORT_DIR)\n # - create subdirectories of .panzer\n subdirs = ['preflight',\n 'filter',\n 'lua-filter',\n 'postprocess',\n 'postflight',\n 'cleanup',\n 'template',\n 'styles']\n for subdir in subdirs:\n target = os.path.join(const.DEFAULT_SUPPORT_DIR, subdir)\n os.mkdir(target)\n info.log('INFO', 'panzer', 'created \"%s\"' % target)\n # - create styles.yaml\n style_definitions = os.path.join(const.DEFAULT_SUPPORT_DIR,\n 'styles',\n 'styles.yaml')\n open(style_definitions, 'w').close()\n info.log('INFO', 'panzer', 'created empty \"styles/styles.yaml\"')\n\ndef resolve_path(filename, kind, options):\n \"\"\" return path to filename of kind field \"\"\"\n basename = os.path.splitext(filename)[0]\n paths = list()\n paths.append(filename)\n paths.append(os.path.join(kind, filename))\n paths.append(os.path.join(kind, basename, filename))\n paths.append(os.path.join(options['panzer']['panzer_support'], kind,\n filename))\n paths.append(os.path.join(options['panzer']['panzer_support'], kind,\n basename,\n filename))\n for path in paths:\n if os.path.exists(path):\n return path\n return filename\n\n" }, { "path": "panzer/version.py", "content": "\"\"\" version of panzer \"\"\"\nVERSION = \"1.4.1\"\n" }, { "path": "setup.py", "content": "# encoding: utf-8\n\nfrom setuptools import setup\nfrom panzer.version import VERSION\n\nwith open('README.rst', 'r', encoding='utf8') as file:\n readme_text = file.read()\n\nsetup(name='panzer',\n version=VERSION,\n description='pandoc with styles',\n long_description=readme_text,\n url='https://github.com/msprev/panzer',\n author='Mark Sprevak',\n author_email='mark.sprevak@ed.ac.uk',\n license='LICENSE.txt',\n packages=['panzer'],\n install_requires=['pandocfilters'],\n include_package_data=True,\n keywords=['pandoc'],\n classifiers=[\n 'Development Status :: 4 - Beta',\n 'Environment :: Console',\n 'Intended Audience :: End Users/Desktop',\n 'Intended Audience :: Developers',\n 'License :: OSI Approved :: BSD License',\n 'Operating System :: OS Independent',\n 'Programming Language :: Python :: 3',\n 'Topic :: Text Processing'\n ],\n entry_points = {\n 'console_scripts': [\n 'panzer = panzer.panzer:main'\n ]\n },\n zip_safe=False)\n" } ]