[
  {
    "path": ".gitignore",
    "content": "/*\n!/*.json\n!/src\n!/Cargo.toml\n!/.gitignore\n!/README.md\n\n"
  },
  {
    "path": "Cargo.toml",
    "content": "[package]\nname = \"fzero\"\nversion = \"0.1.0\"\nauthors = [\"Brandon Falk <bfalk@gamozolabs.com>\"]\nedition = \"2018\"\nlicense = \"MIT\"\n\n# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html\n\n[dependencies]\nserde = { version = \"1.0\", features = [\"derive\"] }\nserde_json = \"1.0\"\n\n"
  },
  {
    "path": "README.md",
    "content": "# Intro\n\n`fzero` is a grammar-based fuzzer that generates a Rust application inspired\nby the paper \"Building Fast Fuzzers\" by Rahul Gopinath and Andreas Zeller.\nhttps://arxiv.org/pdf/1911.07707.pdf\n\nYou can find the F1 fuzzer here:\n\nhttps://github.com/vrthra/F1\n\n# Usage\n\nCurrently this only generates an application that does benchmarking, but with\nsome quick hacks you could easily get the input out and feed it to an\napplication.\n\n## Example usage\n\n```\nD:\\dev\\fzero_fuzz>cargo run --release html.json test.rs test.exe 8\n    Finished release [optimized] target(s) in 0.02s\n     Running `target\\release\\fzero.exe html.json test.rs test.exe 8`\nLoaded grammar json\nConverted grammar to binary format\nOptimized grammar\nGenerated Rust source file\nCreated Rust binary!\n\nD:\\dev\\fzero_fuzz>test.exe\nMiB/sec:    1773.3719\nMiB/sec:    1763.8357\nMiB/sec:    1756.8917\nMiB/sec:    1757.1934\nMiB/sec:    1758.9417\nMiB/sec:    1758.9122\nMiB/sec:    1758.7352\n```\n\n# Concept\n\nThis program takes in an input grammar specified by a JSON file. This JSON\ngrammar representation is converted to a binary-style grammar that is intended\nfor interpretation and optimization. A Rust application (source file) is\nproduced by the shape of the input grammar. This then is compiled using `rustc`\nto an application for the local machine.\n\nThis doesn't have any constraints on the random number generation as it uses an\ninfinite supply of random numbers. There is no limitation on the output size\nand the buffer will dynamically grow as the input is created.\n\n# Benchmarks\n\nAll tests on a single core of a `Intel(R) Core(TM) i7-8700 CPU @ 3.20GHz` with a turbo clock rate of 4.3 GHz\n\nAll numbers in `MiB/second`.\n\n| Benchmark          | fzero fuzzer | F1 fuzzer | Speedup |\n|--------------------|--------------|-----------|---------|\n| html.json depth=4  |         5330 |      1295 |   4.11x |\n| html.json depth=8  |         1760 |       348 |   5.05x |\n| html.json depth=16 |          338 |       195 |   1.73x |\n| html.json depth=32 |          218 |       175 |   1.25x |\n| html.json depth=64 |          201 |       175 |   1.14x |\n| json.json depth=4  |           97 |        97 |   1.00x |\n| json.json depth=8  |           79 |        93 |   0.84x |\n| json.json depth=16 |           83 |        89 |   0.93x |\n| json.json depth=32 |           85 |        88 |   0.97x |\n| json.json depth=64 |           85 |        90 |   0.94x |\n\n# Unsafe code\n\nThis project uses a small amount of `unsafe` code to provide the same semantics\nof `extend_from_slice` but in a much faster way (over 4x faster). Not quite\nsure why it's much faster, but if you are uncomfortable with `unsafe` code,\nfeel free to set `SAFE_ONLY` to `true` at the top of `src/lib.rs`. This will\nrestrict this fuzzer to only generate safe code. I don't think this is\nnecessary but who knows :)\n\n# Performance\n\nThe performance of this tool is separated into multiple categories. One is the\ncode generation side, how long it takes for the JSON to be compiled into a Rust\napplication. The other is the code execution speeds, which is how fast the\nproduced application can generate inputs.\n\n## Code Generation\n\nCode generation vastly outperforms the \"Building Fast Fuzzers\" paper. For\nexample when generating the code based on the `html.json` grammar, the F1\nfuzzer took over 25 minutes to produce the code. This fuzzer is capable of\nproducing a Rust application in under 10 seconds.\n\n## Code execution\n\nThis project is on some performance metrics about 20-30% slower than the F1\nfuzzer, but these scenarios are rare. However, in most situations we've been\nabout to out-perform F1 by about 30-50%, and in extreme cases (html.json\ndepth=8) we've observed over a 4x speedup.\n\n# Differences from the F1 fuzzer\n\nThe F1 fuzzer mentions a technique that will resolve to the nearest terminal\ntokens when stack depth is exceeded. We haven't implemented this technique but\nI don't think it's a huge impact on the generated inputs. This is something I\nwill look into in the future.\n\nDue to not using globals this can easily be scaled out to multiple threads as\nall random state and input generation are done in a structure.\n\nThere is no use of assembly in this project, and thus it can produce\nhighly-performant fuzzers for any architecture or environment that Rust can\ncompile against (pretty much identical to LLVM's target list).\n\n"
  },
  {
    "path": "html.json",
    "content": "{\n    \"<start>\": [[\"<_l_>\", \"!DOCTYPE html\", \"<_r_>\", \"<html_document>\"]],\n    \"<_l_>\": [[\"<\"]],\n    \"<_r_>\": [[\">\"]],\n    \"<_cl_>\": [[\"</\"]],\n    \"<a_tag>\": [[\"<_l_>\", \"a\", \"<d>\", \"<_r_>\", \"<a_content-1>\", \"<_cl_>\", \"a\", \"<_r_>\"]],\n    \"<a_content>\": [[\"<heading>\"], [\"<text>\"]],\n    \"<abbr_tag>\": [[\"<_l_>\", \"abbr\", \"<d>\", \"<_r_>\", \"<text>\", \"<_cl_>\", \"abbr\", \"<_r_>\"]],\n    \"<acronym_tag>\": [[\"<_l_>\", \"acronym\", \"<d>\", \"<_r_>\", \"<text>\", \"<_cl_>\", \"acronym\", \"<_r_>\"]],\n    \"<address_tag>\": [[\"<_l_>\", \"address\", \"<d>\", \"<_r_>\", \"<address_content-1>\", \"<_cl_>\", \"address\", \"<_r_>\"]],\n    \"<address_content>\": [[\"<p_tag>\"], [\"<text>\"]],\n    \"<applet_content>\": [[\"<param-1>\", \"<body_content>\"]],\n    \"<area>\": [[\"<_l_>\", \"area\", \"<d>\", \"<_r_>\"]],\n    \"<applet_tag>\": [[\"<_l_>\", \"applet\", \"<d>\", \"<_r_>\", \"<applet_content>\", \"<_cl_>\", \"applet\", \"<_r_>\"]],\n    \"<b_tag>\": [[\"<_l_>\", \"b\", \"<d>\", \"<_r_>\", \"<text>\", \"<_cl_>\", \"b\", \"<_r_>\"]],\n    \"<basefont_tag>\": [[\"<_l_>\", \"basefront\", \"<d>\", \"<_r_>\", \"<body_content>\", \"<_cl_>\", \"basefront\", \"<_r_>\"]],\n    \"<bdo_tag>\": [[\"<_l_>\", \"bdo\", \"<d>\", \"<_r_>\", \"<text>\", \"<_cl_>\", \"bdo\", \"<_r_>\"]],\n    \"<big_tag>\": [[\"<_l_>\", \"big\", \"<d>\", \"<_r_>\", \"<text>\", \"<_cl_>\", \"big\", \"<_r_>\"]],\n    \"<blink_tag>\": [[\"<_l_>\", \"blink\", \"<d>\", \"<_r_>\", \"<text>\", \"<_cl_>\", \"blink\", \"<_r_>\"]],\n    \"<block>\": [[\"<block_content-1>\"]],\n    \"<block_content>\": [[\"<basefont_tag>\"], [\"<blockquote_tag>\"], [\"<center_tag>\"], [\"<dir_tag>\"], [\"<div_tag>\"], [\"<dl_tag>\"], [\"<form_tag>\"], [\"<listing_tag>\"], [\"<menu_tag>\"], [\"<multicol_tag>\"], [\"<nobr_tag>\"], [\"<ol_tag>\"], [\"<p_tag>\"], [\"<pre_tag>\"], [\"<table_tag>\"], [\"<ul_tag>\"], [\"<xmp_tag>\"]],\n    \"<blockquote_tag>\": [[\"<_l_>\", \"blockquote\", \"<d>\", \"<_r_>\", \"<body_content>\", \"<_cl_>\", \"blockquote\", \"<_r_>\"]],\n    \"<body_content>\": [[\"<_l_>\", \"bgsound\", \"<d>\", \"<_r_>\"], [\"<_l_>\", \"hr\", \"<_r_>\"], [\"<address_tag>\"], [\"<block>\"], [\"<del_tag>\"], [\"<heading>\"], [\"<ins_tag>\"], [\"<layer_tag>\"], [\"<map_tag>\"], [\"<marquee_tag>\"], [\"<text>\"]],\n    \"<body_tag>\": [[\"<_l_>\", \"body\", \"<d>\", \"<_r_>\", \"<body_content-1>\", \"<_cl_>\", \"body\", \"<_r_>\"]],\n    \"<caption_tag>\": [[\"<_l_>\", \"caption\", \"<d>\", \"<_r_>\", \"<body_content-2>\", \"<_cl_>\", \"caption\", \"<_r_>\"]],\n    \"<center_tag>\": [[\"<_l_>\", \"center\", \"<d>\", \"<_r_>\", \"<body_content-3>\", \"<_cl_>\", \"center\", \"<_r_>\"]],\n    \"<cite_tag>\": [[\"<_l_>\", \"cite\", \"<d>\", \"<_r_>\", \"<text>\", \"<_cl_>\", \"cite\", \"<_r_>\"]],\n    \"<code_tag>\": [[\"<_l_>\", \"code\", \"<d>\", \"<_r_>\", \"<text>\", \"<_cl_>\", \"code\", \"<_r_>\"]],\n    \"<colgroup_content>\": [[\"<_l_>\", \"col\", \"<d>\", \"<_r_-1>\"]],\n    \"<colgroup_tag>\": [[\"<_l_>\", \"colgroup\", \"<d>\", \"<_r_>\", \"<colgroup_content>\"]],\n    \"<content_style>\": [[\"<abbr_tag>\"], [\"<acronym_tag>\"], [\"<cite_tag>\"], [\"<code_tag>\"], [\"<dfn_tag>\"], [\"<em_tag>\"], [\"<kbd_tag>\"], [\"<q_tag>\"], [\"<strong_tag>\"], [\"<var_tag>\"]],\n    \"<dd_tag>\": [[\"<_l_>\", \"dd\", \"<d>\", \"<_r_>\", \"<flow>\", \"<_cl_>\", \"dd\", \"<_r_>\"]],\n    \"<del_tag>\": [[\"<_l_>\", \"del\", \"<d>\", \"<_r_>\", \"<flow>\", \"<_cl_>\", \"del\", \"<_r_>\"]],\n    \"<dfn_tag>\": [[\"<_l_>\", \"dfn\", \"<d>\", \"<_r_>\", \"<text>\", \"<_cl_>\", \"dfn\", \"<_r_>\"]],\n    \"<dir_tag>\": [[\"<_l_>\", \"dir\", \"<d>\", \"<_r_>\", \"<li_tag-1>\", \"<_cl_>\", \"dir\", \"<_r_>\"]],\n    \"<div_tag>\": [[\"<_l_>\", \"div\", \"<d>\", \"<_r_>\", \"<body_content>\", \"<_cl_>\", \"div\", \"<_r_>\"]],\n    \"<dl_content>\": [[\"<dt_tag>\", \"<dd_tag>\"]],\n    \"<dl_tag>\": [[\"<_l_>\", \"dl\", \"<d>\", \"<_r_>\", \"<dl_content-1>\", \"<_cl_>\", \"dl\", \"<_r_>\"]],\n    \"<dt_tag>\": [[\"<_l_>\", \"dt\", \"<d>\", \"<_r_>\", \"<text>\", \"<_cl_>\", \"dt\", \"<_r_>\"]],\n    \"<em_tag>\": [[\"<_l_>\", \"em\", \"<d>\", \"<_r_>\", \"<text>\", \"<_cl_>\", \"em\", \"<_r_>\"]],\n    \"<fieldset_tag>\": [[\"<_l_>\", \"fieldset\", \"<d>\", \"<_r_>\", \"<legend_tag-1>\", \"<form_content-1>\", \"<_cl_>\", \"fieldset\", \"<_r_>\"]],\n    \"<flow>\": [[\"<flow_content-1>\"]],\n    \"<flow_content>\": [[\"<block>\"], [\"<text>\"]],\n    \"<font_tag>\": [[\"<_l_>\", \"font\", \"<d>\", \"<_r_>\", \"<style_text>\", \"<_cl_>\", \"font\", \"<_r_>\"]],\n    \"<form_content>\": [[\"<_l_>\", \"input\", \"<d>\", \"<_r_>\"], [\"<_l_>\", \"keygen\", \"<d>\", \"<_r_>\"], [\"<body_content>\"], [\"<fieldset_tag>\"], [\"<label_tag>\"], [\"<select_tag>\"], [\"<textarea_tag>\"]],\n    \"<form_tag>\": [[\"<_l_>\", \"form\", \"<d>\", \"<_r_>\", \"<form_content-2>\", \"<_cl_>\", \"form\", \"<_r_>\"]],\n    \"<frameset_content>\": [[\"<_l_>\", \"frame\", \"<d>\", \"<_r_>\"], [\"<noframes_tag>\"]],\n    \"<frameset_tag>\": [[\"<_l_>\", \"frameset\", \"<d>\", \"<_r_>\", \"<frameset_content-1>\", \"<_cl_>\", \"frameset\", \"<_r_>\"]],\n    \"<h1_tag>\": [[\"<_l_>\", \"h1\", \"<d>\", \"<_r_>\", \"<text>\", \"<_cl_>\", \"h1\", \"<_r_>\"]],\n    \"<h2_tag>\": [[\"<_l_>\", \"h2\", \"<d>\", \"<_r_>\", \"<text>\", \"<_cl_>\", \"h2\", \"<_r_>\"]],\n    \"<h3_tag>\": [[\"<_l_>\", \"h3\", \"<d>\", \"<_r_>\", \"<text>\", \"<_cl_>\", \"h3\", \"<_r_>\"]],\n    \"<h4_tag>\": [[\"<_l_>\", \"h4\", \"<d>\", \"<_r_>\", \"<text>\", \"<_cl_>\", \"h4\", \"<_r_>\"]],\n    \"<h5_tag>\": [[\"<_l_>\", \"h5\", \"<d>\", \"<_r_>\", \"<text>\", \"<_cl_>\", \"h5\", \"<_r_>\"]],\n    \"<h6_tag>\": [[\"<_l_>\", \"h6\", \"<d>\", \"<_r_>\", \"<text>\", \"<_cl_>\", \"h6\", \"<_r_>\"]],\n    \"<head_content>\": [[\"<_l_>\", \"base\", \"<d>\", \"<_r_>\"], [\"<_l_>\", \"link\", \"<d>\", \"<_r_>\"], [\"<_l_>\", \"meta\", \"<d>\", \"<_r_>\"], [\"<style_tag>\"], [\"<title_tag>\"], [\"<script_tag>\"]],\n    \"<head_tag>\": [[\"<_l_>\", \"head\", \"<d>\", \"<_r_>\", \"<head_content-1>\", \"<_cl_>\", \"head\", \"<_r_>\"]],\n    \"<heading>\": [[\"<h1_tag>\"], [\"<h2_tag>\"], [\"<h3_tag>\"], [\"<h4_tag>\"], [\"<h5_tag>\"], [\"<h6_tag>\"]],\n    \"<html_content>\": [[\"<head_tag>\", \"<body_tag>\"], [\"<head_tag>\", \"<frameset_tag>\"]],\n    \"<html_document>\": [[\"<html_tag>\"]],\n    \"<html_tag>\": [[\"<_l_>\", \"html\", \"<_r_>\", \"<html_content>\", \"<_cl_>\", \"html\", \"<_r_>\"]],\n    \"<i_tag>\": [[\"<_l_>\", \"i\", \"<d>\", \"<_r_>\", \"<text>\", \"<_cl_>\", \"i\", \"<_r_>\"]],\n    \"<ilayer_tag>\": [[\"<_l_>\", \"ilayer\", \"<d>\", \"<_r_>\", \"<body_content>\", \"<_cl_>\", \"ilayer\", \"<_r_>\"]],\n    \"<ins_tag>\": [[\"<_l_>\", \"ins\", \"<d>\", \"<_r_>\", \"<flow>\", \"<_cl_>\", \"ins\", \"<_r_>\"]],\n    \"<kbd_tag>\": [[\"<_l_>\", \"kbd\", \"<d>\", \"<_r_>\", \"<text>\", \"<_cl_>\", \"kbd\", \"<_r_>\"]],\n    \"<label_content>\": [[\"<_l_>\", \"input\", \"<d>\", \"<_r_>\"], [\"<body_content>\"], [\"<select_tag>\"], [\"<textarea_tag>\"]],\n    \"<label_tag>\": [[\"<_l_>\", \"label\", \"<d>\", \"<_r_>\", \"<label_content-1>\", \"<_cl_>\", \"label\", \"<_r_>\"]],\n    \"<layer_tag>\": [[\"<_l_>\", \"layer\", \"<d>\", \"<_r_>\", \"<body_content>\", \"<_cl_>\", \"layer\", \"<_r_>\"]],\n    \"<legend_tag>\": [[\"<_l_>\", \"legend\", \"<d>\", \"<_r_>\", \"<text>\", \"<_cl_>\", \"legend\", \"<_r_>\"]],\n    \"<li_tag>\": [[\"<_l_>\", \"li\", \"<d>\", \"<_r_>\", \"<flow>\", \"<_cl_>\", \"li\", \"<_r_>\"]],\n    \"<literal_text>\": [[\"<plain_text>\"]],\n    \"<listing_tag>\": [[\"<_l_>\", \"listing\", \"<d>\", \"<_r_>\", \"<literal_text>\", \"<_cl_>\", \"listing\", \"<_r_>\"]],\n    \"<map_content>\": [[\"<area-1>\"]],\n    \"<map_tag>\": [[\"<_l_>\", \"map\", \"<d>\", \"<_r_>\", \"<map_content>\", \"<_cl_>\", \"map\", \"<_r_>\"]],\n    \"<marquee_tag>\": [[\"<_l_>\", \"marquee\", \"<d>\", \"<_r_>\", \"<style_text>\", \"<_cl_>\", \"marquee\", \"<_r_>\"]],\n    \"<menu_tag>\": [[\"<_l_>\", \"menu\", \"<d>\", \"<_r_>\", \"<li_tag-2>\", \"<_cl_>\", \"menu\", \"<_r_>\"]],\n    \"<multicol_tag>\": [[\"<_l_>\", \"multicol\", \"<d>\", \"<_r_>\", \"<body_content>\", \"<_cl_>\", \"multicol\", \"<_r_>\"]],\n    \"<nobr_tag>\": [[\"<_l_>\", \"nobr\", \"<d>\", \"<_r_>\", \"<text>\", \"<_cl_>\", \"nobr\", \"<_r_>\"]],\n    \"<noembed_tag>\": [[\"<_l_>\", \"noembed\", \"<d>\", \"<_r_>\", \"<text>\", \"<_cl_>\", \"noembed\", \"<_r_>\"]],\n    \"<noframes_tag>\": [[\"<_l_>\", \"noframes\", \"<d>\", \"<_r_>\", \"<body_content-4>\", \"<_cl_>\", \"noframes\", \"<_r_>\"]],\n    \"<noscript_tag>\": [[\"<_l_>\", \"noscript\", \"<d>\", \"<_r_>\", \"<text>\", \"<_cl_>\", \"noscript\", \"<_r_>\"]],\n    \"<object_content>\": [[\"<param-2>\", \"<body_content>\"]],\n    \"<object_tag>\": [[\"<_l_>\", \"object\", \"<d>\", \"<_r_>\", \"<object_content>\", \"<_cl_>\", \"object\", \"<_r_>\"]],\n    \"<ol_tag>\": [[\"<_l_>\", \"ol\", \"<d>\", \"<_r_>\", \"<li_tag-3>\", \"<_cl_>\", \"ol\", \"<_r_>\"]],\n    \"<optgroup_tag>\": [[\"<_l_>\", \"optgroup\", \"<d>\", \"<_r_>\", \"<option_tag-1>\", \"<_cl_>\", \"optgroup\", \"<_r_>\"]],\n    \"<option_tag>\": [[\"<_l_>\", \"option\", \"<d>\", \"<_r_>\", \"<plain_text-1>\", \"<_cl_>\", \"option\", \"<_r_>\"]],\n    \"<p_tag>\": [[\"<_l_>\", \"p\", \"<_r_>\", \"<text>\", \"<_cl_>\", \"p\", \"<_r_>\"]],\n    \"<param>\": [[\"<_l_>\", \"param\", \"<_r_>\"]],\n    \"<plain_text>\": [[\"<entity-1>\"]],\n    \"<entity>\": [[\"<char>\"], [\"<ampersand>\"]],\n    \"<char>\": [[\"7\"], [\"*\"], [\":\"], [\"]\"], [\"n\"], [\"m\"], [\"N\"], [\"/\"], [\".\"], [\"K\"], [\"T\"], [\"I\"], [\"f\"], [\"o\"], [\",\"], [\"l\"], [\"W\"], [\"-\"], [\"?\"], [\"\\\\\"], [\"%\"], [\"1\"], [\"c\"], [\"H\"], [\"!\"], [\"A\"], [\"$\"], [\"9\"], [\"q\"], [\"[\"], [\")\"], [\" \"], [\";\"], [\"b\"], [\"i\"], [\"L\"], [\"'\"], [\"Y\"], [\"\\t\"], [\"3\"], [\"g\"], [\"F\"], [\"E\"], [\"D\"], [\"C\"], [\"@\"], [\"t\"], [\"R\"], [\"\\\"\"], [\"2\"], [\"}\"], [\"~\"], [\"5\"], [\"4\"], [\"z\"], [\"X\"], [\"S\"], [\"O\"], [\"v\"], [\"J\"], [\"`\"], [\"B\"], [\"\\n\"], [\"y\"], [\"p\"], [\"6\"], [\"0\"], [\"k\"], [\"w\"], [\"\\r\"], [\"V\"], [\"_\"], [\"s\"], [\"x\"], [\"{\"], [\"d\"], [\"a\"], [\"#\"], [\"Q\"], [\"<\"], [\"u\"], [\"r\"], [\"U\"], [\"h\"], [\">\"], [\"(\"], [\"P\"], [\"G\"], [\"\\f\"], [\"Z\"], [\"j\"], [\"|\"], [\"e\"], [\"^\"], [\"=\"], [\"8\"], [\"+\"], [\"M\"]],\n    \"<ampersand>\": [[\"&nbsp;\"]],\n    \"<physical_style>\": [[\"<b_tag>\"], [\"<bdo_tag>\"], [\"<big_tag>\"], [\"<blink_tag>\"], [\"<font_tag>\"], [\"<i_tag>\"], [\"<s_tag>\"], [\"<small_tag>\"], [\"<span_tag>\"], [\"<strike_tag>\"], [\"<sub_tag>\"], [\"<sup_tag>\"], [\"<tt_tag>\"], [\"<u_tag>\"]],\n    \"<pre_content>\": [[\"<_l_>\", \"br\", \"<_r_>\"], [\"<_l_>\", \"hr\", \"<_r_>\"], [\"<a_tag>\"], [\"<style_text>\"]],\n    \"<pre_tag>\": [[\"<_l_>\", \"pre\", \"<_r_>\", \"<pre_content-1>\", \"<_cl_>\", \"pre\", \"<_r_>\"]],\n    \"<q_tag>\": [[\"<_l_>\", \"q\", \"<_r_>\", \"<text>\", \"<_cl_>\", \"q\", \"<_r_>\"]],\n    \"<s_tag>\": [[\"<_l_>\", \"s\", \"<_r_>\", \"<text>\", \"<_cl_>\", \"s\", \"<_r_>\"]],\n    \"<script_tag>\": [[\"<_l_>\", \"script\", \"<d>\", \"<_r_>\", \"<plain_text>\", \"<_cl_>\", \"script\", \"<_r_>\"]],\n    \"<select_content>\": [[\"<optgroup_tag>\"], [\"<option_tag>\"]],\n    \"<select_tag>\": [[\"<_l_>\", \"select\", \"<d>\", \"<_r_>\", \"<select_content-1>\", \"<_cl_>\", \"select\", \"<_r_>\"]],\n    \"<small_tag>\": [[\"<_l_>\", \"small\", \"<d>\", \"<_r_>\", \"<text>\", \"<_cl_>\", \"small\", \"<_r_>\"]],\n    \"<span_tag>\": [[\"<_l_>\", \"span\", \"<d>\", \"<_r_>\", \"<text>\", \"<_cl_>\", \"span\", \"<_r_>\"]],\n    \"<strike_tag>\": [[\"<_l_>\", \"strike\", \"<d>\", \"<_r_>\", \"<text>\", \"<_cl_>\", \"strike\", \"<_r_>\"]],\n    \"<strong_tag>\": [[\"<_l_>\", \"strong\", \"<d>\", \"<_r_>\", \"<text>\", \"<_cl_>\", \"strong\", \"<_r_>\"]],\n    \"<style_tag>\": [[\"<_l_>\", \"style\", \"<d>\", \"<_r_>\", \"<plain_text>\", \"<_cl_>\", \"style\", \"<_r_>\"]],\n    \"<style_text>\": [[\"<plain_text>\"]],\n    \"<sub_tag>\": [[\"<_l_>\", \"sub\", \"<d>\", \"<_r_>\", \"<text>\", \"<_cl_>\", \"sub\", \"<_r_>\"]],\n    \"<sup_tag>\": [[\"<_l_>\", \"sup\", \"<d>\", \"<_r_>\", \"<text>\", \"<_cl_>\", \"sup\", \"<_r_>\"]],\n    \"<table_cell>\": [[\"<td_tag>\"], [\"<th_tag>\"]],\n    \"<table_content>\": [[\"<_l_>\", \"tbody\", \"<d>\", \"<_r_>\"], [\"<_l_>\", \"tfoot\", \"<d>\", \"<_r_>\"], [\"<_l_>\", \"thead\", \"<d>\", \"<_r_>\"], [\"<tr_tag>\"]],\n    \"<table_tag>\": [[\"<_l_>\", \"table\", \"<d>\", \"<_r_>\", \"<caption_tag-1>\", \"<colgroup_tag-1>\", \"<table_content-1>\", \"<_cl_>\", \"table\", \"<_r_>\"]],\n    \"<td_tag>\": [[\"<_l_>\", \"td\", \"<d>\", \"<_r_>\", \"<body_content>\", \"<_cl_>\", \"td\", \"<_r_>\"]],\n    \"<text>\": [[\"<text_content-1>\"]],\n    \"<text_content>\": [[\"<_l_>\", \"br\", \"<d>\", \"<_r_>\"], [\"<_l_>\", \"embed\", \"<d>\", \"<_r_>\"], [\"<_l_>\", \"iframe\", \"<d>\", \"<_r_>\"], [\"<_l_>\", \"img\", \"<d>\", \"<_r_>\"], [\"<_l_>\", \"spacer\", \"<d>\", \"<_r_>\"], [\"<_l_>\", \"wbr\", \"<d>\", \"<_r_>\"], [\"<a_tag>\"], [\"<applet_tag>\"], [\"<content_style>\"], [\"<ilayer_tag>\"], [\"<noembed_tag>\"], [\"<noscript_tag>\"], [\"<object_tag>\"], [\"<plain_text>\"], [\"<physical_style>\"]],\n    \"<textarea_tag>\": [[\"<_l_>\", \"textarea\", \"<d>\", \"<_r_>\", \"<plain_text>\", \"<_cl_>\", \"textarea\", \"<_r_>\"]],\n    \"<th_tag>\": [[\"<_l_>\", \"th\", \"<d>\", \"<_r_>\", \"<body_content>\", \"<_cl_>\", \"th\", \"<_r_>\"]],\n    \"<title_tag>\": [[\"<_l_>\", \"title\", \"<d>\", \"<_r_>\", \"<plain_text>\", \"<_cl_>\", \"title\", \"<_r_>\"]],\n    \"<tr_tag>\": [[\"<_l_>\", \"tr\", \"<d>\", \"<_r_>\", \"<table_cell-1>\", \"<_cl_>\", \"tr\", \"<_r_>\"]],\n    \"<tt_tag>\": [[\"<_l_>\", \"tt\", \"<d>\", \"<_r_>\", \"<text>\", \"<_cl_>\", \"tt\", \"<_r_>\"]],\n    \"<u_tag>\": [[\"<_l_>\", \"u\", \"<d>\", \"<_r_>\", \"<text>\", \"<_cl_>\", \"u\", \"<_r_>\"]],\n    \"<ul_tag>\": [[\"<_l_>\", \"ul\", \"<d>\", \"<_r_>\", \"<li_tag-4>\", \"<_cl_>\", \"ul\", \"<_r_>\"]],\n    \"<var_tag>\": [[\"<_l_>\", \"var\", \"<d>\", \"<_r_>\", \"<text>\", \"<_cl_>\", \"var\", \"<_r_>\"]],\n    \"<xmp_tag>\": [[\"<_l_>\", \"xmp\", \"<d>\", \"<_r_>\", \"<literal_text>\", \"<_cl_>\", \"xmp\", \"<_r_>\"]],\n    \"<d>\": [[\"<space-1>\", \"<attributes-1>\", \"<space-2>\"], []],\n    \"<attribute>\": [[\"<key>\"], [\"<key>\", \"=\\\"\", \"<value>\", \"\\\"\"], [\"<key>\", \"='\", \"<value>\", \"'\"], [\"<key>\", \"=\", \"<uqvalue>\"]],\n    \"<key>\": [[\"<allchars>\"]],\n    \"<allchars>\": [[\"7\"], [\"*\"], [\":\"], [\"&\"], [\"]\"], [\"n\"], [\"m\"], [\"N\"], [\".\"], [\"K\"], [\"T\"], [\"I\"], [\"f\"], [\"o\"], [\",\"], [\"l\"], [\"W\"], [\"-\"], [\"?\"], [\"\\\\\"], [\"%\"], [\"1\"], [\"c\"], [\"H\"], [\"!\"], [\"A\"], [\"$\"], [\"9\"], [\"q\"], [\"[\"], [\")\"], [\";\"], [\"b\"], [\"i\"], [\"L\"], [\"Y\"], [\"3\"], [\"g\"], [\"F\"], [\"E\"], [\"D\"], [\"C\"], [\"@\"], [\"t\"], [\"R\"], [\"2\"], [\"}\"], [\"~\"], [\"5\"], [\"4\"], [\"z\"], [\"X\"], [\"S\"], [\"O\"], [\"v\"], [\"J\"], [\"`\"], [\"B\"], [\"y\"], [\"p\"], [\"6\"], [\"0\"], [\"k\"], [\"w\"], [\"\\r\"], [\"V\"], [\"_\"], [\"s\"], [\"x\"], [\"{\"], [\"d\"], [\"a\"], [\"#\"], [\"Q\"], [\"u\"], [\"r\"], [\"U\"], [\"h\"], [\"(\"], [\"P\"], [\"G\"], [\"\\f\"], [\"Z\"], [\"j\"], [\"|\"], [\"e\"], [\"^\"], [\"8\"], [\"+\"], [\"M\"]],\n    \"<value>\": [[\"<anychars>\"]],\n    \"<anychar>\": [[\"0\"], [\"1\"], [\"2\"], [\"3\"], [\"4\"], [\"5\"], [\"6\"], [\"7\"], [\"8\"], [\"9\"], [\"a\"], [\"b\"], [\"c\"], [\"d\"], [\"e\"], [\"f\"], [\"g\"], [\"h\"], [\"i\"], [\"j\"], [\"k\"], [\"l\"], [\"m\"], [\"n\"], [\"o\"], [\"p\"], [\"q\"], [\"r\"], [\"s\"], [\"t\"], [\"u\"], [\"v\"], [\"w\"], [\"x\"], [\"y\"], [\"z\"], [\"A\"], [\"B\"], [\"C\"], [\"D\"], [\"E\"], [\"F\"], [\"G\"], [\"H\"], [\"I\"], [\"J\"], [\"K\"], [\"L\"], [\"M\"], [\"N\"], [\"O\"], [\"P\"], [\"Q\"], [\"R\"], [\"S\"], [\"T\"], [\"U\"], [\"V\"], [\"W\"], [\"X\"], [\"Y\"], [\"Z\"], [\"!\"], [\"\\\"\"], [\"#\"], [\"$\"], [\"%\"], [\"&\"], [\"'\"], [\"(\"], [\")\"], [\"*\"], [\"+\"], [\",\"], [\"-\"], [\".\"], [\"/\"], [\":\"], [\";\"], [\"<\"], [\"=\"], [\">\"], [\"?\"], [\"@\"], [\"[\"], [\"\\\\\"], [\"]\"], [\"^\"], [\"_\"], [\"`\"], [\"{\"], [\"|\"], [\"}\"], [\"~\"], [\" \"], [\"\\t\"], [\"\\n\"], [\"\\r\"], [\"\\u000b\"], [\"\\f\"]],\n    \"<anychars>\": [[\"<anychar-1>\"]],\n    \"<uqvalue>\": [[\"<uqchars>\"]],\n    \"<uqchar>\": [[\"7\"], [\"*\"], [\":\"], [\"&\"], [\"]\"], [\"n\"], [\"m\"], [\"N\"], [\".\"], [\"K\"], [\"T\"], [\"I\"], [\"f\"], [\"o\"], [\",\"], [\"l\"], [\"W\"], [\"-\"], [\"?\"], [\"\\\\\"], [\"%\"], [\"1\"], [\"c\"], [\"H\"], [\"!\"], [\"A\"], [\"$\"], [\"9\"], [\"q\"], [\"[\"], [\")\"], [\";\"], [\"b\"], [\"i\"], [\"L\"], [\"Y\"], [\"3\"], [\"g\"], [\"F\"], [\"E\"], [\"D\"], [\"C\"], [\"@\"], [\"t\"], [\"R\"], [\"2\"], [\"}\"], [\"~\"], [\"5\"], [\"4\"], [\"z\"], [\"X\"], [\"S\"], [\"O\"], [\"v\"], [\"J\"], [\"B\"], [\"y\"], [\"p\"], [\"6\"], [\"0\"], [\"k\"], [\"w\"], [\"\\r\"], [\"V\"], [\"_\"], [\"s\"], [\"x\"], [\"{\"], [\"d\"], [\"a\"], [\"#\"], [\"Q\"], [\"u\"], [\"r\"], [\"U\"], [\"h\"], [\"(\"], [\"P\"], [\"G\"], [\"\\f\"], [\"Z\"], [\"j\"], [\"|\"], [\"e\"], [\"^\"], [\"8\"], [\"+\"], [\"M\"]],\n    \"<uqchars>\": [[\"<uqchar-1>\"]],\n    \"<attributes>\": [[\"<attribute>\"], [\"<attribute>\", \"<space-3>\", \"<attributes>\"]],\n    \"<space>\": [[\" \"], [\"\\t\"], [\"\\n\"]],\n    \"<a_content-1>\": [[], [\"<a_content>\", \"<a_content-1>\"]],\n    \"<address_content-1>\": [[], [\"<address_content>\", \"<address_content-1>\"]],\n    \"<param-1>\": [[], [\"<param>\", \"<param-1>\"]],\n    \"<block_content-1>\": [[], [\"<block_content>\", \"<block_content-1>\"]],\n    \"<body_content-1>\": [[], [\"<body_content>\", \"<body_content-1>\"]],\n    \"<body_content-2>\": [[], [\"<body_content>\", \"<body_content-2>\"]],\n    \"<body_content-3>\": [[], [\"<body_content>\", \"<body_content-3>\"]],\n    \"<_r_-1>\": [[], [\"<_r_>\", \"<_r_-1>\"]],\n    \"<li_tag-1>\": [[\"<li_tag>\"], [\"<li_tag>\", \"<li_tag-1>\"]],\n    \"<dl_content-1>\": [[\"<dl_content>\"], [\"<dl_content>\", \"<dl_content-1>\"]],\n    \"<legend_tag-1>\": [[], [\"<legend_tag>\", \"<legend_tag-1>\"]],\n    \"<form_content-1>\": [[], [\"<form_content>\", \"<form_content-1>\"]],\n    \"<flow_content-1>\": [[], [\"<flow_content>\", \"<flow_content-1>\"]],\n    \"<form_content-2>\": [[], [\"<form_content>\", \"<form_content-2>\"]],\n    \"<frameset_content-1>\": [[], [\"<frameset_content>\", \"<frameset_content-1>\"]],\n    \"<head_content-1>\": [[], [\"<head_content>\", \"<head_content-1>\"]],\n    \"<label_content-1>\": [[], [\"<label_content>\", \"<label_content-1>\"]],\n    \"<area-1>\": [[], [\"<area>\", \"<area-1>\"]],\n    \"<li_tag-2>\": [[], [\"<li_tag>\", \"<li_tag-2>\"]],\n    \"<body_content-4>\": [[], [\"<body_content>\", \"<body_content-4>\"]],\n    \"<param-2>\": [[], [\"<param>\", \"<param-2>\"]],\n    \"<li_tag-3>\": [[\"<li_tag>\"], [\"<li_tag>\", \"<li_tag-3>\"]],\n    \"<option_tag-1>\": [[], [\"<option_tag>\", \"<option_tag-1>\"]],\n    \"<plain_text-1>\": [[\"<plain_text>\"], [\"<plain_text>\", \"<plain_text-1>\"]],\n    \"<entity-1>\": [[], [\"<entity>\", \"<entity-1>\"]],\n    \"<pre_content-1>\": [[], [\"<pre_content>\", \"<pre_content-1>\"]],\n    \"<select_content-1>\": [[], [\"<select_content>\", \"<select_content-1>\"]],\n    \"<caption_tag-1>\": [[], [\"<caption_tag>\", \"<caption_tag-1>\"]],\n    \"<colgroup_tag-1>\": [[], [\"<colgroup_tag>\", \"<colgroup_tag-1>\"]],\n    \"<table_content-1>\": [[], [\"<table_content>\", \"<table_content-1>\"]],\n    \"<text_content-1>\": [[], [\"<text_content>\", \"<text_content-1>\"]],\n    \"<table_cell-1>\": [[], [\"<table_cell>\", \"<table_cell-1>\"]],\n    \"<li_tag-4>\": [[], [\"<li_tag>\", \"<li_tag-4>\"]],\n    \"<space-1>\": [[\"<space>\"], [\"<space>\", \"<space-1>\"]],\n    \"<attributes-1>\": [[], [\"<attributes>\", \"<attributes-1>\"]],\n    \"<space-2>\": [[], [\"<space>\", \"<space-2>\"]],\n    \"<anychar-1>\": [[], [\"<anychar>\", \"<anychar-1>\"]],\n    \"<uqchar-1>\": [[\"<uqchar>\"], [\"<uqchar>\", \"<uqchar-1>\"]],\n    \"<space-3>\": [[\"<space>\"], [\"<space>\", \"<space-3>\"]]\n}\n"
  },
  {
    "path": "json.json",
    "content": "{\n    \"<start>\": [[\"<json>\"]],\n    \"<json>\": [[\"<element>\"]],\n    \"<element>\": [[\"<ws>\", \"<value>\", \"<ws>\"]],\n    \"<value>\": [[\"<object>\"], [\"<array>\"], [\"<string>\"], [\"<number>\"],\n                [\"true\"], [\"false\"],\n                [\"null\"]],\n    \"<object>\": [[\"{\", \"<ws>\", \"}\"], [\"{\", \"<members>\", \"}\"]],\n    \"<members>\": [[\"<member>\", \"<symbol-2>\"]],\n    \"<member>\": [[\"<ws>\", \"<string>\", \"<ws>\", \":\", \"<element>\"]],\n    \"<array>\": [[\"[\", \"<ws>\", \"]\"], [\"[\", \"<elements>\", \"]\"]],\n    \"<elements>\": [[\"<element>\", \"<symbol-1-1>\"]],\n    \"<string>\": [[\"\\\"\", \"<characters>\", \"\\\"\"]],\n    \"<characters>\": [[\"<character-1>\"]],\n    \"<character>\": [[\"0\"], [\"1\"], [\"2\"], [\"3\"], [\"4\"], [\"5\"], [\"6\"], [\"7\"],\n                    [\"8\"], [\"9\"], [\"a\"], [\"b\"], [\"c\"], [\"d\"], [\"e\"], [\"f\"],\n                    [\"g\"], [\"h\"], [\"i\"], [\"j\"], [\"k\"], [\"l\"], [\"m\"], [\"n\"],\n                    [\"o\"], [\"p\"], [\"q\"], [\"r\"], [\"s\"], [\"t\"], [\"u\"], [\"v\"],\n                    [\"w\"], [\"x\"], [\"y\"], [\"z\"], [\"A\"], [\"B\"], [\"C\"], [\"D\"],\n                    [\"E\"], [\"F\"], [\"G\"], [\"H\"], [\"I\"], [\"J\"], [\"K\"], [\"L\"],\n                    [\"M\"], [\"N\"], [\"O\"], [\"P\"], [\"Q\"], [\"R\"], [\"S\"], [\"T\"],\n                    [\"U\"], [\"V\"], [\"W\"], [\"X\"], [\"Y\"], [\"Z\"], [\"!\"], [\"#\"],\n                    [\"$\"], [\"%\"], [\"&\"], [\"\\\"\"], [\"(\"], [\")\"], [\"*\"], [\"+\"],\n                    [\",\"], [\"-\"], [\".\"], [\"/\"], [\":\"], [\";\"], [\"<\"], [\"=\"],\n                    [\">\"], [\"?\"], [\"@\"], [\"[\"], [\"]\"], [\"^\"], [\"_\"], [\"`\"],\n                    [\"{\"], [\"|\"], [\"}\"], [\"~\"], [\" \"], [\"<esc>\"]],\n    \"<esc>\": [[\"\\\\\",\"<escc>\"]],\n    \"<escc>\": [[\"\\\\\"],[\"b\"],[\"f\"], [\"n\"], [\"r\"],[\"t\"],[\"\\\"\"]],\n    \"<number>\": [[\"<int>\", \"<frac>\", \"<exp>\"]],\n    \"<int>\": [[\"<digit>\"], [\"<onenine>\", \"<digits>\"], [\"-\", \"<digits>\"],\n              [\"-\", \"<onenine>\", \"<digits>\"]],\n    \"<digits>\": [[\"<digit-1>\"]],\n    \"<digit>\": [[\"0\"], [\"<onenine>\"]],\n    \"<onenine>\": [[\"1\"], [\"2\"], [\"3\"], [\"4\"], [\"5\"], [\"6\"], [\"7\"], [\"8\"],\n                  [\"9\"]],\n    \"<frac>\": [[], [\".\", \"<digits>\"]],\n    \"<exp>\": [[], [\"E\", \"<sign>\", \"<digits>\"], [\"e\", \"<sign>\", \"<digits>\"]],\n    \"<sign>\": [[], [\"+\"], [\"-\"]],\n    \"<ws>\": [[\"<sp1>\", \"<ws>\"], []],\n    \"<sp1>\": [[\" \"],[\"\\n\"],[\"\\t\"],[\"\\r\"]],\n    \"<symbol>\": [[\",\", \"<members>\"]],\n    \"<symbol-1>\": [[\",\", \"<elements>\"]],\n    \"<symbol-2>\": [[], [\"<symbol>\", \"<symbol-2>\"]],\n    \"<symbol-1-1>\": [[], [\"<symbol-1>\", \"<symbol-1-1>\"]],\n    \"<character-1>\": [[], [\"<character>\", \"<character-1>\"]],\n    \"<digit-1>\": [[\"<digit>\"], [\"<digit>\", \"<digit-1>\"]]\n}\n"
  },
  {
    "path": "src/main.rs",
    "content": "use std::collections::{BTreeMap, BTreeSet};\nuse std::path::Path;\nuse std::process::Command;\nuse serde::{Deserialize, Serialize};\n\n/// If this is `true` then the output file we generate will not emit any\n/// unsafe code. I'm not aware of any bugs with the unsafe code that I use and\n/// thus this is by default set to `false`. Feel free to set it to `true` if\n/// you are concerned.\nconst SAFE_ONLY: bool = false;\n\n/// Representation of a grammar file in a Rust structure. This allows us to\n/// use Serde to serialize and deserialize the json grammar files\n#[derive(Serialize, Deserialize, Default, Debug)]\nstruct Grammar(BTreeMap<String, Vec<Vec<String>>>);\n\n/// A strongly typed wrapper around a `usize` which selects different fragment\n/// identifiers\n#[derive(Clone, Copy, Debug)]\nstruct FragmentId(usize);\n\n/// A fragment which is specified by the grammar file\n#[derive(Clone, Debug)]\nenum Fragment {\n    /// A non-terminal fragment which refers to a list of `FragmentId`s to\n    /// randomly select from for expansion\n    NonTerminal(Vec<FragmentId>),\n\n    /// A list of `FragmentId`s that should be expanded in order\n    Expression(Vec<FragmentId>),\n\n    /// A terminal fragment which simply should expand directly to the\n    /// contained vector of bytes\n    Terminal(Vec<u8>),\n\n    /// A fragment which does nothing. This is used during optimization passes\n    /// to remove fragments with no effect.\n    Nop,\n}\n\n/// A grammar representation in Rust that is designed to be easy to work with\n/// in-memory and optimized for code generation.\n#[derive(Debug, Default)]\nstruct GrammarRust {\n    /// All types\n    fragments: Vec<Fragment>,\n\n    /// Cached fragment identifier for the start node\n    start: Option<FragmentId>,\n\n    /// Mapping of non-terminal names to fragment identifers\n    name_to_fragment: BTreeMap<String, FragmentId>,\n}\n\nimpl GrammarRust {\n    /// Create a new Rust version of a `Grammar` which was loaded via a\n    /// grammar json specification.\n    fn new(grammar: &Grammar) -> Self {\n        // Create a new grammar structure\n        let mut ret = GrammarRust::default();\n\n        // Parse the input grammar to resolve all fragment names\n        for (non_term, _) in grammar.0.iter() {\n            // Make sure that there aren't duplicates of fragment names\n            assert!(!ret.name_to_fragment.contains_key(non_term),\n                \"Duplicate non-terminal definition, fail\");\n\n            // Create a new, empty fragment\n            let fragment_id = ret.allocate_fragment(\n                Fragment::NonTerminal(Vec::new()));\n\n            // Add the name resolution for the fragment\n            ret.name_to_fragment.insert(non_term.clone(), fragment_id);\n        }\n\n        // Parse the input grammar\n        for (non_term, fragments) in grammar.0.iter() {\n            // Get the non-terminal fragment identifier\n            let fragment_id = ret.name_to_fragment[non_term];\n\n            // Create a vector to hold all of the variants possible under this\n            // non-terminal fragment\n            let mut variants = Vec::new();\n\n            // Go through all sub-fragments\n            for js_sub_fragment in fragments {\n                // Different options for this sub-fragment\n                let mut options = Vec::new();\n\n                // Go through each option in the sub-fragment\n                for option in js_sub_fragment {\n                    let fragment_id = if let Some(&non_terminal) =\n                            ret.name_to_fragment.get(option) {\n                        // If we can resolve the name of this fragment, it is a\n                        // non-terminal fragment and should be allocated as\n                        // such\n                        ret.allocate_fragment(\n                            Fragment::NonTerminal(vec![non_terminal]))\n                    } else {\n                        // Convert the terminal bytes into a vector and\n                        // create a new fragment containing it\n                        ret.allocate_fragment(Fragment::Terminal(\n                            option.as_bytes().to_vec()))\n                    };\n\n                    // Push this fragment as an option\n                    options.push(fragment_id);\n                }\n\n                // Create a new fragment of all the options\n                variants.push(\n                    ret.allocate_fragment(Fragment::Expression(options)));\n            }\n\n            // Get access to the fragment we want to update based on the\n            // possible variants\n            let fragment = &mut ret.fragments[fragment_id.0];\n\n            // Overwrite the terminal definition\n            *fragment = Fragment::NonTerminal(variants);\n        }\n\n        // Resolve the start node\n        ret.start = Some(ret.name_to_fragment[\"<start>\"]);\n\n        ret\n    }\n\n    /// Allocate a new fragment identifier and add it to the fragment list\n    pub fn allocate_fragment(&mut self, fragment: Fragment) -> FragmentId {\n        // Get a unique fragment identifier\n        let fragment_id = FragmentId(self.fragments.len());\n\n        // Store the fragment\n        self.fragments.push(fragment);\n\n        fragment_id\n    }\n\n    /// Optimize to remove fragments with non-random effects\n    pub fn optimize(&mut self) {\n        // Keeps track of fragment identifiers which resolve to nops\n        let mut nop_fragments = BTreeSet::new();\n\n        // Track if a optimization had an effect\n        let mut changed = true;\n        while changed {\n            // Start off assuming no effect from optimzation\n            changed = false;\n\n            // Go through each fragment, looking for potential optimizations\n            for idx in 0..self.fragments.len() {\n                // Clone the fragment such that we can inspect it, but we also\n                // can mutate it in place.\n                match self.fragments[idx].clone() {\n                    Fragment::NonTerminal(options) => {\n                        // If this non-terminal only has one option, replace\n                        // itself with the only option it resolves to\n                        if options.len() == 1 {\n                            self.fragments[idx] =\n                                self.fragments[options[0].0].clone();\n                            changed = true;\n                        }\n                    }\n                    Fragment::Expression(expr) => {\n                        // If this expression doesn't have anything to do at\n                        // all. Then simply replace it with a `Nop`\n                        if expr.len() == 0 {\n                            self.fragments[idx] = Fragment::Nop;\n                            changed = true;\n\n                            // Track that this fragment identifier now resolves\n                            // to a nop\n                            nop_fragments.insert(idx);\n                        }\n\n                        // If this expression only does one thing, then replace\n                        // the expression with the thing that it does.\n                        if expr.len() == 1 {\n                            self.fragments[idx] =\n                                self.fragments[expr[0].0].clone();\n                            changed = true;\n                        }\n\n                        // Remove all `Nop`s from this expression, as they\n                        // wouldn't result in anything occuring.\n                        if let Fragment::Expression(exprs) =\n                                &mut self.fragments[idx] {\n                            // Only retain fragments which are not nops\n                            exprs.retain(|x| {\n                                if nop_fragments.contains(&x.0) {\n                                    // Fragment was a nop, remove it\n                                    changed = true;\n                                    false\n                                } else {\n                                    // Fragment was fine, keep it\n                                    true\n                                }\n                            });\n                        }\n                    }\n                    Fragment::Terminal(_) | Fragment::Nop => {\n                        // Already maximally optimized\n                    }\n                }\n            }\n        }\n    }\n\n    /// Generate a new Rust program that can be built and will generate random\n    /// inputs and benchmark them\n    pub fn program<P: AsRef<Path>>(&self, path: P, max_depth: usize) {\n        let mut program = String::new();\n\n        // Construct the base of the application. This is a profiling loop that\n        // is used for testing.\n        program += &format!(r#\"\n#![allow(unused)]\nuse std::cell::Cell;\nuse std::time::Instant;\n\nfn main() {{\n    let mut fuzzer = Fuzzer {{\n        seed:  Cell::new(0x34cc028e11b4f89c),\n        buf:   Vec::new(),\n    }};\n    \n    let mut generated = 0usize;\n    let it = Instant::now();\n\n    for iters in 1u64.. {{\n        fuzzer.buf.clear();\n        fuzzer.fragment_{}(0);\n        generated += fuzzer.buf.len();\n\n        // Filter to reduce the amount of times printing occurs\n        if (iters & 0xfffff) == 0 {{\n            let elapsed = (Instant::now() - it).as_secs_f64();\n            let bytes_per_sec = generated as f64 / elapsed;\n            print!(\"MiB/sec: {{:12.4}}\\n\", bytes_per_sec / 1024. / 1024.);\n        }}\n    }}\n}}\n\nstruct Fuzzer {{\n    seed:  Cell<usize>,\n    buf:   Vec<u8>,\n}}\n\nimpl Fuzzer {{\n    fn rand(&self) -> usize {{\n        let mut seed = self.seed.get();\n        seed ^= seed << 13;\n        seed ^= seed >> 17;\n        seed ^= seed << 43;\n        self.seed.set(seed);\n        seed\n    }}\n\"#, self.start.unwrap().0);\n\n        // Go through each fragment in the list of fragments\n        for (id, fragment) in self.fragments.iter().enumerate() {\n            // Create a new function for this fragment\n            program += &format!(\"    fn fragment_{}(&mut self, depth: usize) {{\\n\", id);\n\n            // Add depth checking to terminate on depth exhaustion\n            program += &format!(\"        if depth >= {} {{ return; }}\\n\",\n                max_depth);\n\n            match fragment {\n                Fragment::NonTerminal(options) => {\n                    // For non-terminal cases pick a random variant to select\n                    // and invoke that fragment's routine\n                    program += &format!(\"        match self.rand() % {} {{\\n\", options.len());\n\n                    for (option_id, option) in options.iter().enumerate() {\n                        program += &format!(\"            {} => self.fragment_{}(depth + 1),\\n\", option_id, option.0);\n                    }\n                    program += &format!(\"            _ => unreachable!(),\\n\");\n\n                    program += &format!(\"        }}\\n\");\n                }\n                Fragment::Expression(expr) => {\n                    // Invoke all of the expression's routines in order\n                    for &exp in expr.iter() {\n                        program += &format!(\"        self.fragment_{}(depth + 1);\\n\", exp.0);\n                    }\n                }\n                Fragment::Terminal(value) => {\n                    // Append the terminal value to the output buffer\n                    if SAFE_ONLY {\n                        program += &format!(\"        self.buf.extend_from_slice(&{:?});\\n\",\n                            value);\n                    } else {\n                        // For some reason this is faster than\n                        // `extend_from_slice` even though it does the exact\n                        // same thing. This was observed to be over a 4-5x\n                        // speedup in some scenarios.\n                        program += &format!(r#\"\n            unsafe {{\n                let old_size = self.buf.len();\n                let new_size = old_size + {};\n\n                if new_size > self.buf.capacity() {{\n                    self.buf.reserve(new_size - old_size);\n                }}\n\n                std::ptr::copy_nonoverlapping({:?}.as_ptr(), self.buf.as_mut_ptr().offset(old_size as isize), {});\n                self.buf.set_len(new_size);\n            }}\n    \"#, value.len(), value, value.len());\n                    }\n                }\n                Fragment::Nop => {}\n            }\n\n            program += \"    }\\n\";\n        }\n        program += \"}\\n\";\n\n        // Write out the test application\n        std::fs::write(path, program)\n            .expect(\"Failed to create output Rust application\");\n    }\n}\n\nfn main() -> std::io::Result<()> {\n    // Get access to the command line arguments\n    let args: Vec<String> = std::env::args().collect();\n    if args.len() != 5 {\n        print!(\"usage: fzero <grammar json> <output Rust file> <output binary name> <max depth>\\n\");\n        return Ok(());\n    }\n\n    // Load up a grammar file\n    let grammar: Grammar = serde_json::from_slice(\n        &std::fs::read(&args[1])?)?;\n    print!(\"Loaded grammar json\\n\");\n\n    // Convert the grammar file to the Rust structures\n    let mut gram = GrammarRust::new(&grammar);\n    print!(\"Converted grammar to binary format\\n\");\n\n    // Optimize the grammar\n    gram.optimize();\n    print!(\"Optimized grammar\\n\");\n\n    // Generate a Rust application\n    gram.program(&args[2],\n        args[4].parse().expect(\"Invalid digit in max depth\"));\n    print!(\"Generated Rust source file\\n\");\n\n    // Compile the application\n    // rustc -O -g test.rs -C target-cpu=native\n    let status = Command::new(\"rustc\")\n        .arg(\"-O\")                // Optimize the binary\n        .arg(\"-g\")                // Generate debug information\n        .arg(&args[2])            // Name of the input Rust file\n        .arg(\"-C\")                // Optimize for the current microarchitecture\n        .arg(\"target-cpu=native\")\n        .arg(\"-o\")                // Output filename\n        .arg(&args[3]).spawn()?.wait()?;\n    assert!(status.success(), \"Failed to compile Rust binary\");\n    print!(\"Created Rust binary!\\n\");\n\n    Ok(())\n}\n\n"
  }
]